README
Github:
https://github.com/baaihealth/opi
Paper:
OPI: An Open Instruction Dataset for Adapting Large Language Models to Protein-Related Tasks has been accepted by NeurIPS 2024 Workshop: Foundation Models for Science: Progress, Opportunities, and Challenges.
Model Card of OPI-Galactica-6.7B
OPI-Galactica-6.7B was fine-tuned from the Galactica-6.7B model using the complete OPI training set (i.e.,OPI_full_1.61M_train.json). For more details of training and testing, please visit https://github.com/baaihealth/opi.
Evaluation of OPI-Galactica-6.7B model on 9 tasks
Each testing result is derived from the Galactica-6.7B model that has been fine-tuned using OPI_full_1.61M.json and subsequently evaluated on the respective testing set for each specific task.
| Task Type | Task Name | Testing file | Accuracy | Precision | Recall | F1 | Rouge-L |
|---|---|---|---|---|---|---|---|
| Sequence Understanding | EC Number Prediction (split100) | CLEAN_EC_number_new_test | - | 0.2700 | 0.2663 | 0.2596 | - |
| CLEAN_EC_number_price_test | - | 0.0268 | 0.0268 | 0.0268 | - | ||
| Fold Type Prediction | fold_type_test_Fold_Holdout | 0.0808 | - | - | - | - | |
| fold_type_test_Superfamily_Holdout | 0.1348 | - | - | - | - | ||
| fold_type_test_Family_Holdout | 0.4854 | - | - | - | - | ||
| Subcellular Localization Prediction | subcell_loc_test | 0.7771 | - | - | - | - | |
| Annotation Prediction | Function Keywords Prediction | CASPSimilarSeq_keywords_test | - | 0.8120 | 0.7360 | 0.7643 | - |
| Function Keywords Prediction | IDFilterSeq_keywords_test | - | 0.8377 | 0.8019 | 0.8070 | - | |
| Function Keywords Prediction | UniProtSeq_keywords_test | - | 0.8596 | 0.8196 | 0.8276 | - | |
| Gene Ontology (GO) Terms Prediction | CASPSimilarSeq_go_terms_test | - | 0.7613 | 0.7492 | 0.7476 | - | |
| Gene Ontology (GO) Terms Prediction | IDFilterSeq_go_terms_test | - | 0.7404 | 0.7274 | 0.7207 | - | |
| Gene Ontology (GO) Terms Prediction | UniProtSeq_go_terms_test | - | 0.7638 | 0.7373 | 0.7358 | - | |
| Function Description Prediction | CASPSimilarSeq_function_test | - | - | - | - | 0.7430 | |
| Function Description Prediction | IDFilterSeq_function_test | - | - | - | - | 0.7014 | |
| Function Description Prediction | UniProtSeq_function_test | - | - | - | - | 0.7133 | |
| Knowledge Mining | Tissue Location Prediction from Gene Symbol | gene_symbol_to_tissue_test | - | 0.3917 | 0.9077 | 0.5303 | - |
| Cancer Prediction from Gene Symbol | gene_symbol_to_cancer_test | - | 0.3555 | 0.3189 | 0.3229 | - | |
| Cancer Prediction from Gene Name | gene_name_to_cancer_test | - | 0.2728 | 0.2554 | 0.2533 | - |
Prediction comparison with SOTA mdoels
Demo
We use the FastChat platform to visually demonstrate the ability of OPI-Galactica-6.7B model on various evaluation tasks.
Similar repos
README
Github:
https://github.com/baaihealth/opi
Paper:
OPI: An Open Instruction Dataset for Adapting Large Language Models to Protein-Related Tasks has been accepted by NeurIPS 2024 Workshop: Foundation Models for Science: Progress, Opportunities, and Challenges.
Model Card of OPI-Galactica-6.7B
OPI-Galactica-6.7B was fine-tuned from the Galactica-6.7B model using the complete OPI training set (i.e.,OPI_full_1.61M_train.json). For more details of training and testing, please visit https://github.com/baaihealth/opi.
Evaluation of OPI-Galactica-6.7B model on 9 tasks
Each testing result is derived from the Galactica-6.7B model that has been fine-tuned using OPI_full_1.61M.json and subsequently evaluated on the respective testing set for each specific task.
| Task Type | Task Name | Testing file | Accuracy | Precision | Recall | F1 | Rouge-L |
|---|---|---|---|---|---|---|---|
| Sequence Understanding | EC Number Prediction (split100) | CLEAN_EC_number_new_test | - | 0.2700 | 0.2663 | 0.2596 | - |
| CLEAN_EC_number_price_test | - | 0.0268 | 0.0268 | 0.0268 | - | ||
| Fold Type Prediction | fold_type_test_Fold_Holdout | 0.0808 | - | - | - | - | |
| fold_type_test_Superfamily_Holdout | 0.1348 | - | - | - | - | ||
| fold_type_test_Family_Holdout | 0.4854 | - | - | - | - | ||
| Subcellular Localization Prediction | subcell_loc_test | 0.7771 | - | - | - | - | |
| Annotation Prediction | Function Keywords Prediction | CASPSimilarSeq_keywords_test | - | 0.8120 | 0.7360 | 0.7643 | - |
| Function Keywords Prediction | IDFilterSeq_keywords_test | - | 0.8377 | 0.8019 | 0.8070 | - | |
| Function Keywords Prediction | UniProtSeq_keywords_test | - | 0.8596 | 0.8196 | 0.8276 | - | |
| Gene Ontology (GO) Terms Prediction | CASPSimilarSeq_go_terms_test | - | 0.7613 | 0.7492 | 0.7476 | - | |
| Gene Ontology (GO) Terms Prediction | IDFilterSeq_go_terms_test | - | 0.7404 | 0.7274 | 0.7207 | - | |
| Gene Ontology (GO) Terms Prediction | UniProtSeq_go_terms_test | - | 0.7638 | 0.7373 | 0.7358 | - | |
| Function Description Prediction | CASPSimilarSeq_function_test | - | - | - | - | 0.7430 | |
| Function Description Prediction | IDFilterSeq_function_test | - | - | - | - | 0.7014 | |
| Function Description Prediction | UniProtSeq_function_test | - | - | - | - | 0.7133 | |
| Knowledge Mining | Tissue Location Prediction from Gene Symbol | gene_symbol_to_tissue_test | - | 0.3917 | 0.9077 | 0.5303 | - |
| Cancer Prediction from Gene Symbol | gene_symbol_to_cancer_test | - | 0.3555 | 0.3189 | 0.3229 | - | |
| Cancer Prediction from Gene Name | gene_name_to_cancer_test | - | 0.2728 | 0.2554 | 0.2533 | - |
Prediction comparison with SOTA mdoels
Demo
We use the FastChat platform to visually demonstrate the ability of OPI-Galactica-6.7B model on various evaluation tasks.