GatorTronS overview

Developed by a joint effort between the University of Florida and NVIDIA, GatorTronS is a clinical language model of 345 million parameters, pre-trained using a BERT architecure implemented in the Megatron package (https://github.com/NVIDIA/Megatron-LM).

GatorTronS is pre-trained using a dataset consisting of:

• 22B synthetic clinical words generated by GatorTronGPT (a Megatron GPT-3 model)
• 6.1B words from PubMed CC0,
• 2.5B words from WikiText,
• 0.5B words of de-identified clinical notes from MIMIC-III

The Github for GatorTronGPT is at : https://github.com/uf-hobi-informatics-lab/GatorTronGPT

This model is converted to Hugginface from : https://catalog.ngc.nvidia.com/orgs/nvidia/teams/clara/models/gatortron_s

22B synthetic clinical text description

We sampled the beginning 15 tokens from all sections of the de-identified notes from the MIMIC III database and generated approximately 8 million prompts. We also tried several random seeds in GatorTronGPT to generate multiple documents from one prompt. We controlled GatorTronGPT to generate a maximum length of 512 tokens. We apply GatorTronGPT to generate a total of 22 billion words of synthetic clinical text. Detailed information is provided in the GatorTronGPT paper: https://arxiv.org/abs/2305.13523

Model variations

How to use

from transformers import AutoModel, AutoTokenizer, AutoConfig

tokinizer= AutoTokenizer.from_pretrained('UFNLP/gatortronS')
config=AutoConfig.from_pretrained('UFNLP/gatortronS')
mymodel=AutoModel.from_pretrained('UFNLP/gatortronS')

encoded_input=tokinizer("Bone scan:  Negative for distant metastasis.", return_tensors="pt")
encoded_output = mymodel(**encoded_input)
print (encoded_output)

• An NLP pacakge using GatorTronS for clinical concept extraction (Named Entity Recognition): https://github.com/uf-hobi-informatics-lab/ClinicalTransformerNER
• An NLP pacakge using GatorTronS for Relation Extraction: https://github.com/uf-hobi-informatics-lab/ClinicalTransformerRelationExtraction
• An NLP pacakge using GatorTronS for extraction of social determinants of health (SDoH) from clinical narratives: https://github.com/uf-hobi-informatics-lab/SDoH_SODA

Citation info

Peng C, Yang X, Chen A, Smith KE, PourNejatian N, Costa AB, Martin C, Flores MG, Zhang Y, Magoc T, Lipori G, Mitchell DA, Ospina NS, Ahmed MM, Hogan WR, Shenkman EA, Guo Y, Bian J, Wu Y†. A Study of Generative Large Language Model for Medical Research and Healthcare. 2023; https://arxiv.org/abs/2305.13523.

• BibTeX entry

@ARTICLE{Peng2023-sm,
   title         = "A study of generative large language model for medical
                    research and healthcare",
   author        = "Peng, Cheng and Yang, Xi and Chen, Aokun and Smith, Kaleb E
                    and PourNejatian, Nima and Costa, Anthony B and Martin,
                    Cheryl and Flores, Mona G and Zhang, Ying and Magoc, Tanja
                    and Lipori, Gloria and Mitchell, Duane A and Ospina, Naykky
                    S and Ahmed, Mustafa M and Hogan, William R and Shenkman,
                    Elizabeth A and Guo, Yi and Bian, Jiang and Wu, Yonghui",
   month         =  may,
   year          =  2023,
   copyright     = "http://arxiv.org/licenses/nonexclusive-distrib/1.0/",
   archivePrefix = "arXiv",
   primaryClass  = "cs.CL",
   eprint        = "2305.13523"
 }
 

Contact

• Yonghui Wu: yonghui.wu@ufl.edu
• Cheng Peng: c.peng@ufl.edu