Fine-Tuning Foundational Models To Code Diagnoses From Veterinary Health Records

The primary purpose of this study is to develop a fast and reliable LLM-based tool for the automatic diagnosis coding of Colorado State University (CSU) veterinary medical summaries to 7,739 SNOMED-CT codes. To achieve this, a variety of human and veterinary clinical, and non-clinical foundational LLMs were fine-tuned on a labeled training dataset curated by expert manual clinical coders at the CSU Veterinary Teaching Hospital. This study was largely inspired by VetTag (2019), an automatic veterinary diagnosis coding tool developed by researchers at Stanford University using a subset of the CSU labeled training dataset. It was of interest to improve upon the original model by implementing better training practices, considering a wider variety of models, and employing the model for more practically significant tasks.

Abstract

Veterinary medical records represent a large data resource for application to veterinary and One Health clinical research efforts. Use of the data is limited by interoperability challenges including inconsistent data formats and data siloing. Clinical coding using standardized medical terminologies enhances the quality of medical records and facilitates their interoperability with veterinary and human health records from other sites. Previous studies, such as DeepTag and VetTag, have evaluated the application of Natural Language Processing (NLP) to automate veterinary diagnosis coding, employing long short-term memory (LSTM) and transformer models to infer a subset of Systemized Nomenclature of Medicine - Clinical Terms (SNOMED-CT) diagnosis codes from free-text clinical notes. This study expands on these efforts by incorporating all 7,739 distinct SNOMED-CT diagnosis codes recognized by the Colorado State University (CSU) Veterinary Teaching Hospital (VTH) and by leveraging the increasing availability of pre-trained large language models (LLMs). Ten freely-available pre-trained LLMs (GatorTron, MedicalAI ClinicalBERT, medAlpaca, VetBERT, PetBERT, BERT, BERT Large, RoBERTa, GPT-2, and GPT-2 XL) were fine-tuned on the free-text notes from 246,473 manually-coded veterinary patient visits included in the CSU VTH's electronic health records (EHR), which resulted in superior performance relative to previous efforts. The most accurate results were obtained when expansive labeled data were used to fine-tune relatively large clinical LLMs, but the study also showed that comparable results can be obtained using more limited resources and non-clinical LLMs. The results of this study contribute to the improvement of the quality of veterinary EHR by investigating accessible methods for automated coding and support both animal and human health research by paving the way for more integrated and comprehensive health databases that span species and institutions.

Approach

Existing state-of-the-art human and veterinary clinical LLMs and general knowledge LLMs were leveraged by fine-tuning them for veterinary clinical coding to 7,739 SNOMED-CT diagnoses. All models were fine-tuned under a consistent training framework.

Loss Function: Binary Cross-Entropy with Logits
Optimizer: AdamW
Batch Size: 32
Initial Learning Rate: $5*10^{-8}$
Pleateau Learning Rate: $3*10^{-5}$
Warmup Steps: 5,000
Early Stopping: 5 epoch patience based on validation loss
Maximum Epochs: 50

Usage

Several Python scripts are involved in the fine-tuning pipeline for this study:
preprocess.py: Handles the cleaning of raw clinical text, data preprocessing, and tokenization. Raw text in the form a .csv file is inputted and a set of .pt PyTorch tensors are outputted. Data are split into training/valiation/testing partitions which are saved separately.
models.py: Defines custom PyTorch model classes for each foundational model. The raw model weights are downloaded from HuggingFace, portions of each model are set to allow for fine-tuning, and additional custom layers are appended to each model (e.g., classifier layers).
trainer.py: Executes a fine-tuning procedure using previously preprocessed data and defined model classes. A fine-tuning procedure is initialized by executing a slurm script named as {model name}_(server name).slurm. The configuration of these slurm scripts is custom to CSU's computing resources.
evaluate.py: Performs auxiliary plotting tasks to evaluate model performance after fine-tuning is complete. The functions in this script are called by the fine-tuning code.

Models

Results

Data Availability

The data used to fine-tune foundational models for this study are propriety medical records that are the property of Colorado State University; they cannot be shared publicly. Once proper approvals are obtained, we hope to be able to share fine-tuned model weights publicly under the control of a data usage agreement.

Citation

@software{CSU_NLPEHR_2024,
  author = {Mayla Boguslav and Adam Kiehl and David Kott},
  title = {Fine-Tuning Foundational Models To Code Diagnoses From Veterinary Health Records},
  url = {https://github.com/adam-kiehl/DiagnosisCoding},
  date = {2025-06-17},
  note = {GitHub repository},
  organization = {Colorado State University}
}

@misc{boguslav2024finetuningfoundationalmodelscode,
      title={Fine-tuning foundational models to code diagnoses from veterinary health records}, 
      author={Mayla R. Boguslav and Adam Kiehl and David Kott and G. Joseph Strecker and Tracy Webb and Nadia Saklou and Terri Ward and Michael Kirby},
      year={2024},
      eprint={2410.15186},
      archivePrefix={arXiv},
      primaryClass={cs.CL},
      url={https://arxiv.org/abs/2410.15186}, 
}

References

• Iz Beltagy, Matthew E Peters, and Arman Cohan. “Longformer: The long-document transformer”. In: arXiv preprint arXiv:2004.05150 (2020).
• Marieke M van Buchem et al. “Diagnosis Classification in the Emergency Room Using Natural Language Processing”. In: Caring is Sharing–Exploiting the Value in Data for Health and Innovation. IOS Press, 2023, pp. 815–816.
• Sharon Campbell and Katrina Giadresco. “Computer-assisted clinical coding: A narrative review of the literature on its benefits, limitations, implementation and impact on clinical coding professionals”. In: Health Information Management Journal 49.1 (2020), pp. 5–18.
• Eunsuk Chang and Javed Mostafa. “The use of SNOMED CT, 2013-2020: a literature review”. In: Journal of the American Medical Informatics Association 28.9 (2021), pp. 2017–2026.
• Qingyu Chen et al. “Large language models in biomedical natural language processing: benchmarks, baselines, and recommendations”. In: arXiv preprint arXiv:2305.16326 (2023).
• Zeming Chen et al. MEDITRON-70B: Scaling Medical Pretraining for Large Language Models. 2023. arXiv: 2311.16079 [cs.CL].
• Ha Na Cho et al. “Task-Specific Transformer-Based Language Models in Health Care: Scoping Review”. In: JMIR Medical Informatics 12 (2024), e49724.
• Jacob Devlin et al. “BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding”. In: CoRR abs/1810.04805 (2018). arXiv: 1810.04805. url: http://arxiv.org/abs/1810.04805.
• Centers for Disease Control and Prevention. One Health. https://www.cdc.gov/onehealth/index.html. 2024.
• Hang Dong et al. “Automated clinical coding: what, why, and where we are?” In: NPJ digital medicine 5.1 (2022), p. 159.
• María C Durango, Ever A Torres-Silva, and Andrés Orozco-Duque. “Named Entity Recognition in Electronic Health Records: A Methodological Review”. In: Healthcare Informatics Research 29.4 (2023), p. 286.
• Sean Farrell et al. “PetBERT: automated ICD-11 syndromic disease coding for outbreak detection in first opinion veterinary electronic health records”. In: Scientific Reports 13.1 (2023), p. 18015.
• Jiayi Feng et al. “Automated generation of ICD-11 cluster codes for Precision Medical Record Classification”. In: International Journal of Computers Communications & Control 19.1 (2024).
• Tuankasfee Hama et al. “Enhancing Patient Outcome Prediction through Deep Learning with Sequential Diagnosis Codes from structural EHR: A systematic review”. In: ().
• Tianyu Han et al. “MedAlpaca–An Open-Source Collection of Medical Conversational AI Models and Training Data”. In: arXiv preprint arXiv:2304.08247 (2023).
• Pengcheng He, Jianfeng Gao, and Weizhu Chen. DeBERTaV3: Improving DeBERTa using ELECTRA-Style Pre-Training with Gradient-Disentangled Embedding Sharing. 2021. arXiv: 2111.09543 [cs.CL].
• Historical Association Reference Sets. https://confluence.ihtsdotools.org/display/DOCRELFMT/5.2.5.1+Historical+Association+Reference+Sets. 2024.
• Kexin Huang, Jaan Altosaar, and Rajesh Ranganath. “ClinicalBERT: Modeling Clinical Notes and Predicting Hospital Readmission”. In: arXiv:1904.05342 (2019).
• Brian Hur et al. “Domain adaptation and instance selection for disease syndrome classification over veterinary clinical notes”. In: Proceedings of the 19th SIGBioMed Workshop on Biomedical Language Processing. 2020, pp. 156–166.
• ICD-11. https://icd.who.int/en. 2024.
• Yixing Jiang et al. “VetLLM: Large Language Model for Predicting Diagnosis from Veterinary Notes”. In: PACIFIC SYMPOSIUM ON BIOCOMPUTING 2024. World Scientific. 2023, pp. 120–133.
• Alistair EW Johnson et al. “MIMIC-III, a freely accessible critical care database”. In: Scientific data 3.1 (2016), pp. 1–9.
• Alistair EW Johnson et al. “MIMIC-IV, a freely accessible electronic health record dataset”. In: Scientific data 10.1 (2023), p. 1.
• Ramakanth Kavuluru, Anthony Rios, and Yuan Lu. “An empirical evaluation of supervised learning approaches in assigning diagnosis codes to electronic medical records”. In: Artificial intelligence in medicine 65.2 (2015), pp. 155–166.
• Jianfu Li et al. “A comparative study of pre-trained language models for named entity recognition in clinical trial eligibility criteria from multiple corpora”. In: BMC medical informatics and decision making 22.Suppl 3 (2022), p. 235.
• Yinhan Liu et al. “Roberta: A robustly optimized bert pretraining approach”. In: arXiv preprint arXiv:1907.11692 (2019).
• Shayne Longpre et al. “A pretrainer’s guide to training data: Measuring the effects of data age, domain coverage, quality, & toxicity”. In: Proceedings of the 2024 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies (Volume 1: Long Papers). 2024, pp. 3245–3276.
• Jonathan L Lustgarten et al. “Veterinary informatics: forging the future between veterinary medicine, human medicine, and One Health initiatives—a joint paper by the Association for Veterinary Informatics (AVI) and the CTSA One Health Alliance (COHA)”. In: JAMIA open 3.2 (2020), pp. 306–317.
• Sumit Madan et al. “Transformer models in biomedicine”. In: BMC Medical Informatics and Decision Making 24.1 (2024), p. 214.
• Paul McGreevy et al. “VetCompass Australia: a national big data collection system for veterinary science”. In: Animals 7.10 (2017), p. 74.
• National Library of Medicine. SNOMED CT United States Edition. https://www.nlm.nih.gov/healthit/snomedct/us_edition.html. 2024.
• James Mullenbach et al. “Explainable Prediction of Medical Codes from Clinical Text”. In: Proceedings of the 2018 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long Papers). Ed. by Marilyn Walker, Heng Ji, and Amanda Stent. New Orleans, Louisiana: Association for Computational Linguistics, June 2018, pp. 1101–1111. doi: 10.18653/v1/N18-1100. url: https://aclanthology.org/N18-1100/.
• Sebastian Nagel. “Common Crawl News”. In: URL: http://commoncrawl.org/2016/10/news-datasetavailable/ (visited on 06/20/2020) (cit. on p. 137) (2016).
• Allen Nie et al. “DeepTag: inferring diagnoses from veterinary clinical notes”. In: NPJ digital medicine 1.1 (2018), p. 60.
• Observational Health Data Sciences and Informatics. https://www.ohdsi.org. 2024.
• Nessrine Omrani et al. “To trust or not to trust? An assessment of trust in AI-based systems: Concerns, ethics and contexts”. In: Technological Forecasting and Social Change 181 (2022), p. 121763.
• David Opitz and Richard Maclin. “Popular ensemble methods: An empirical study”. In: Journal of artificial intelligence research 11 (1999), pp. 169–198.
• Zenhwa Ouyang et al. “A scoping review of ‘big data’, ‘informatics’, and ‘bioinformatics’ in the animal health and veterinary medical literature”. In: Animal health research reviews 20.1 (2019), pp. 1–18.
• Ashley N Paynter et al. “Veterinary big data: when data goes to the dogs”. In: Animals 11.7 (2021), p. 1872.
• Joshua Peterson, Stephan Meylan, and David Bourgin. OpenWebText. https://github.com/jcpeterson/openwebtext?tab=readme-ov-file. 2019.
• Nusrat Jahan Prottasha et al. “Parameter-efficient fine-tuning of large language models using semantic knowledge tuning”. In: Scientific Reports 14.1 (2024), p. 30667.
• Alec Radford et al. “Language models are unsupervised multitask learners”. In: OpenAI blog 1.8 (2019), p. 9.
• Philip Resnik and Jimmy Lin. “Evaluation of NLP systems”. In: The handbook of computational linguistics and natural language processing (2010), pp. 271–295.
• Yuqi Si et al. “Deep representation learning of patient data from Electronic Health Records (EHR): A systematic review”. In: Journal of biomedical informatics 115 (2021), p. 103671.
• Emma Strubell, Ananya Ganesh, and Andrew McCallum. “Energy and policy considerations for modern deep learning research”. In: Proceedings of the AAAI conference on artificial intelligence. Vol. 34. 09. 2020, pp. 13693–13696.
• Mukund Sundararajan, Ankur Taly, and Qiqi Yan. “Axiomatic attribution for deep networks”. In: International conference on machine learning. PMLR. 2017, pp. 3319–3328.
• Rohan Taori et al. Stanford Alpaca: An Instruction-following LLaMA model. https://github.com/tatsu-lab/stanford_alpaca. 2023.
• Arun James Thirunavukarasu et al. “Large language models in medicine”. In: Nature medicine 29.8 (2023), pp. 1930–1940.
• Hugo Touvron et al. “Llama 2: Open foundation and fine-tuned chat models”. In: arXiv preprint arXiv:2307.09288 (2023).
• Trieu H Trinh and Quoc V Le. “A simple method for commonsense reasoning”. In: arXiv preprint arXiv:1806.02847 (2018).
• Ashish Vaswani et al. “Attention is All you Need”. In: Advances in Neural Information Processing Systems. Ed. by I. Guyon et al. Vol. 30. Curran Associates, Inc., 2017. url: https://proceedings.neurips.cc/paper_files/paper/2017/file/3f5ee243547dee91fbd053c1c4a845aa-Paper.pdf.
• Guhan Ram Venkataraman et al. “FasTag: Automatic text classification of unstructured medical narratives”. In: PLoS one 15.6 (2020), e0234647.
• Veterinary Nomenclature. https://venomcoding.org. 2024.
• Guangyu Wang et al. “Optimized glycemic control of type 2 diabetes with reinforcement learning: a proof-of-concept trial”. In: Nature Medicine 29.10 (2023), pp. 2633–2642.
• B Warner et al. “Smarter, better, faster, longer: A modern bidirectional encoder for fast, memory efficient, and long context finetuning and inference (2024)”. In: arXiv preprint arXiv.2412.13663 ().
• Who We Are. https://ohdsi.org/who-we-are/. 2024.
• Qianqian Xie et al. “Me-llama: Foundation large language models for medical applications”. In: Research square (2024), rs–3.
• Xi Yang et al. “A large language model for electronic health records”. In: npj Digital Medicine 5.1 (2022), p. 194.
• Yuhui Zhang et al. “VetTag: improving automated veterinary diagnosis coding via large-scale language modeling”. In: NPJ digital medicine 2.1 (2019), p. 35.
• Yukun Zhu et al. “Aligning Books and Movies: Towards Story-like Visual Explanations by Watching Movies and Reading Books”. In: CoRR abs/1506.06724 (2015). arXiv: 1506.06724. url: http://arxiv.org/abs/1506.06724.