SLMs in Healthcare: A Survey

Unlike vanilla contextual pre-trained fundamentally small language models (e.g., ClinicalBERT), our interest lies in compressed and optimized approaches for language models in healthcare, developed as a resource-efficient and domain-specialized solution to LLMs.

Datasets for Healthcare Informatics

1. MIMIC-III, a freely accessible critical care database. Paper | Link
2. PubMedQA: A Dataset for Biomedical Research Question Answering. Paper | Link
3. DrugComb: an integrative cancer drug combination data portal. Paper | Link
4. BI55/MedText, a medical diagnosis dataset containing over 1000 top notch textbook quality patient presentations and diagnosis/treatments. Link
5. keivalya/MedQuad-MedicalQnADataset: A Question-Entailment Approach to Question Answering. Paper | Link
6. AMEGA-Benchmark: Autonomous medical evaluation for guideline adherence of large language models. Paper | Link
7. Medical-Diff-VQA: A Large-Scale Medical Dataset for Difference Visual Question Answering on Chest X-Ray Images. Paper | Link
8. epfl-llm/guidelines: MEDITRON-70B: Scaling Medical Pretraining for Large Language Models. Paper | Link
9. MedMCQA : A Large-scale Multi-Subject Multi-Choice Dataset for Medical domain Question Answering. Paper | Link

SLMs for Healthcare

100M to 5B parameters

1. BioGPT: generative pre-trained transformer for biomedical text generation and mining. Paper | Model
2. BioMedLM: A 2.7B Parameter Language Model Trained On Biomedical Text. Paper | Model
3. RadPhi-2: A Vision-Language Foundation Model to Enhance Efficiency of Chest X-ray Interpretation. Paper | Model
4. RadPhi-3: Small Language Models for Radiology. Paper
5. TinyLLAMA-1.1B: An Open-Source Small Language Model. Paper | Model
6. TinyLlama-1.1B-Medical: A smaller version of https://huggingface.co/therealcyberlord/llama2-qlora-finetuned-medical, which used Llama 2 7B. Model
7. Cura-Llama: Evaluating open-source large language Model’s question answering capability on medical domain. Paper
8. CancerGPT: for few shot drug pair synergy prediction using large pretrained language models. Paper
9. ClinicalMamba: A Generative Clinical Language Model on Longitudinal Clinical Notes. Paper | Model
10. MentalQLM: A lightweight large language model for mental healthcare based on instruction tuning and dual LoRA modules. Paper | Model
11. mhGPT: A Lightweight Generative Pre-Trained Transformer for Mental Health Text Analysis. Paper
12. Med-MoE: Mixture of Domain-Specific Experts for Lightweight Medical Vision-Language Models. Paper | Model
13. HealsHealthAI: Unveiling Personalized Healthcare Insights with Open Source Fine-Tuned LLM. Paper
14. Apollo: A Lightweight Multilingual Medical LLM towards Democratizing Medical AI to 6B People. Paper | Model
15. Med-Pal: Lightweight Large Language Model for Medication Enquiry. Paper

>5B parameters

1. Me-llama: Foundation large language models for medical applications. Paper
2. BioMistral: A Collection of Open-Source Pretrained Large Language Models for Medical Domains. Paper
3. BioMistral-NLU: Towards More Generalizable Medical Language Understanding through Instruction Tuning. Paper | Model
4. Meerkat-7B: Small Language Models Learn Enhanced Reasoning Skills from Medical Textbooks. Paper | Model
5. MentaLLaMA: Interpretable Mental Health Analysis on Social Media with Large Language Models. Paper | Model
6. Med-R^2: Crafting Trustworthy LLM Physicians through Retrieval and Reasoning of Evidence-Based Medicine. Paper | Model
7. ChatDoctor: A Medical Chat Model Fine-Tuned on a Large Language Model Meta-AI (LLaMA) Using Medical Domain Knowledge. Paper | Model
8. MEDITRON-7B: Scaling Medical Pretraining for Large Language Models. Paper | Model
9. PMC-LLaMA: Towards Building Open-source Language Models for Medicine. Paper | Model
10. MedAlpaca: An Open-Source Collection of Medical Conversational AI Models and Training Data. Paper | Model
11. AlpaCare:Instruction-tuned Large Language Models for Medical Application. Paper | Model
12. Psy-LLM: Scaling up Global Mental Health Psychological Services with AI-based Large Language Models. Paper
13. CancerLLM: A Large Language Model in Cancer Domain. Paper

Optimize SLMs for healthcare

Pretraining strategies

1. Large language model benchmarks in medical tasks. Paper
2. Llama-3-Meditron: An Open-Weight Suite of Medical LLMs Based on Llama-3.1 Paper
3. Apollo: A Lightweight Multilingual Medical LLM towards Democratizing Medical AI to 6B People. Paper
4. Large Language Models and Large Multimodal Models in Medical Imaging: A Primer for Physicians. Paper
5. Large language models are poor medical coders—benchmarking of medical code querying. Paper
6. Leveraging large language models for clinical abbreviation disambiguation. Paper

Attention mechanisms

1. Attention is not all you need: the complicated case of ethically using large language models in healthcare and medicine. Paper

Prompt Engineering

1. Pre-train, prompt, and predict: A systematic survey of prompting methods in natural language processing. Paper
2. Prompt engineering for healthcare: Methodologies and applications. Paper
3. Chain of thought utilization in large language models and application in nephrology. Paper
4. ATSCOT: Chain of Thought for Structuring Anesthesia Medical Records. Paper
5. CoMT: Chain-of-medical-thought reduces hallucination in medical report generation. Paper
6. Surgraw: Multi-agent workflow with chain-of-thought reasoning for surgical intelligence. Paper
7. AutoMedPrompt: A New Framework for Optimizing LLM Medical Prompts Using Textual Gradients. Paper

Fine-tuning

1. Do we still need clinical language models? Paper
2. Health Care Language Models and Their Fine-Tuning for Information Extraction: Scoping Review. Paper
3. How to Design, Create, and Evaluate an Instruction-Tuning Dataset for Large Language Model Training in Health Care: Tutorial From a Clinical Perspective. Paper
4. BioInstruct: instruction tuning of large language models for biomedical natural language processing. Paper
5. Instruction Tuning Large Language Models to Understand Electronic Health Records. Paper
6. Medalign: A clinician-generated dataset for instruction following with electronic medical records. Paper
7. LlamaCare: An Instruction Fine-Tuned Large Language Model for Clinical NLP. Paper
8. Alpacare: Instruction-tuned large language models for medical application. Paper
9. LEAP: LLM instruction-example adaptive prompting framework for biomedical relation extraction. Paper
10. Mdagents: An adaptive collaboration of llms for medical decision-making. Paper

Knowledge distillation

1. Distilling the knowledge from large-language model for health event prediction. Paper
2. Large language model distilling medication recommendation model. Paper
3. SleepCoT: A Lightweight Personalized Sleep Health Model via Chain-of-Thought Distillation. Paper
4. Non-small cell lung cancer detection through knowledge distillation approach with teaching assistant. Paper
5. Distilling Large Language Models for Efficient Clinical Information Extraction. Paper
6. LLM-Enhanced Multi-Teacher Knowledge Distillation for Modality-Incomplete Emotion Recognition in Daily Healthcare. Paper
7. Adapt-and-Distill: Developing Small, Fast and Effective Pretrained Language Models for Domains. Paper

Quantization

1. Privacy-Preserving SAM Quantization for Efficient Edge Intelligence in Healthcare. Paper
2. Mental Healthcare Chatbot Based on Custom Diagnosis Documents Using a Quantized Large Language Model. Paper
3. MentalQLM: A lightweight large language model for mental healthcare based on instruction tuning and dual LoRA modules. Paper
4. BioMedLM: A 2.7B Parameter Language Model Trained On Biomedical Text. Paper
5. mhGPT: A Lightweight Generative Pre-Trained Transformer for Mental Health Text Analysis. Paper
6. BioMistral: A Collection of Open-Source Pretrained Large Language Models for Medical Domains. Paper

Pruning

1. All-in-One Tuning and Structural Pruning for Domain-Specific LLMs. Paper
2. Pruning as a Domain-specific LLM Extractor. Paper
3. LLM-Pruner: On the Structural Pruning of Large Language Models. Paper

Reasoning

1. Chain of thought utilization in large language models and application in nephrology. Paper
2. ATSCOT: Chain of Thought for Structuring Anesthesia Medical Records. Paper
3. CoMT: Chain-of-medical-thought reduces hallucination in medical report generation. Paper
4. Surgraw: Multi-agent workflow with chain-of-thought reasoning for surgical intelligence. Paper
5. AutoMedPrompt: A New Framework for Optimizing LLM Medical Prompts Using Textual Gradients. Paper
6. Merging Clinical Knowledge into Large Language Models for Medical Research and Applications: A Survey. Paper
7. AlzheimerRAG: Multimodal Retrieval Augmented Generation for PubMed articles. Paper
8. RT: a Retrieving and Chain-of-Thought framework for few-shot medical named entity recognition. Paper
9. Integrating Chain-of-Thought and Retrieval Augmented Generation Enhances Rare Disease Diagnosis from Clinical Notes. Paper