Awesome-Foundation-Models-for-Advancing-Healthcare

[NEWS.20241115] Our survey paper has been accepted by IEEE Reviews in Biomedical Engineering (IF: 17.2).

[NEWS.20240405] The related survey paper has been released.

[NOTE] If you have any questions, please don't hesitate to contact us.

Foundation model, which is pre-trained on broad data and is able to adapt to a wide range of tasks, is advancing healthcare. It promotes the development of healthcare artificial intelligence (AI) models, breaking the contradiction between limited AI models and diverse healthcare practices. Much more widespread healthcare scenarios will benefit from the development of a healthcare foundation model (HFM), improving their advanced intelligent healthcare services.

This repository is a collection of AWESOME things about Foundation models in healthcare, including language foundation models (LFMs), vision foundation models (VFMs), bioinformatics foundation models (BFMs), and multimodal foundation models (MFMs). Feel free to star and fork.

This repository provides the advancement of current healthcare foundation models based on the following paper:

Foundation Model for Advancing Healthcare: Challenges, Opportunities and Future Directions 中译版
Yuting He, Fuxiang Huang, Xinrui Jiang, Yuxiang Nie, Minghao Wang, Jiguang Wang, Hao Chen
SMART Lab, Hong Kong University of Science and Technology
IEEE Reviews in Biomedical Engineering

If you find our survey beneficial to your work, we would greatly appreciate it if you cite it in your paper:

@ARTICLE{10750441,
  author={He, Yuting and Huang, Fuxiang and Jiang, Xinrui and Nie, Yuxiang and Wang, Minghao and Wang, Jiguang and Chen, Hao},
  journal={IEEE Reviews in Biomedical Engineering}, 
  title={Foundation Model for Advancing Healthcare: Challenges, Opportunities and Future Directions}, 
  year={2024},
  volume={},
  number={},
  pages={1-20},
  doi={10.1109/RBME.2024.3496744}}

Contents

• Awesome-Foundation-Models-for-Advancing-Healthcare
• Related survery
• Methods
  • LFM methods
  • VFM methods
  • BFM methods
  • MFM methods
• Datasets
  • LFM datasets
  • VFM datasets
  • BFM datasets
  • MFM datasets
  • Large-scale comprehensive databases
• Other Resources
  • Lectures and tutorials
  • Blogs
  • Related awesome repositories

Related survey

2024

• [arXiv] Foundation models for biomedical image segmentation: A survey. [Paper]
• [arXiv] Progress and opportunities of foundation models in bioinformatics. [Paper]
• [arXiv] Large language models in bioinformatics: applications and perspectives. [Paper]
• [arXiv] Data-centric foundation models in computational healthcare: A survey. [Paper]
• [arXiv] Vision-Language Models for Medical Report Generation and Visual Question Answering: A Review. [Paper]

2023

• [ACM Computing Surveys] Pre-trained language models in biomedical domain: A systematic survey. [Paper]
• [Nature medicine] Large language models in medicine. [Paper]
• [arXiv] A survey of large language models in medicine: Progress, application, and challenge. [Paper]
• [arXiv] A survey of large language models for healthcare: from data, technology, and applications to accountability and ethics. [Paper]
• [arXiv] Large language models illuminate a progressive pathway to artificial healthcare assistant: A review. [Paper]
• [arXiv] Foundational models in medical imaging: A comprehensive survey and future vision. [Paper]
• [arXiv] CLIP in medical imaging: A comprehensive survey. [Paper]
• [arXiv] Medical vision language pretraining: A survey. [Paper]
• [MIR] Pre-training in medical data: A survey. [Paper]
• [J-BHI] Large AI models in health informatics: Applications, challenges, and the future. [Paper]
• [MedComm–Future Medicine] Accelerating the integration of ChatGPT and other large-scale AI models into biomedical research and healthcare. [Paper]
• [Nature] Foundation models for generalist medical artificial intelligence. [Paper]
• [MedIA] On the challenges and perspectives of foundation models for medical image analysis. [Paper]

Methods

LFM methods

2024

• [AAAI] Zhongjing: Enhancing the chinese medical capabilities of large language model through expert feedback and realworld multi-turn dialogue. [Paper] [Code]
• [NeurIPS] MDAgents: An adaptive collaboration of LLMs for medical decision-making. [Paper] [Code]
• [arXiv] Me LLaMA: Foundation large language models for medical applications [Paper] [Code]
• [arXiv] BioMistral: A collection of open-source pretrained large language models for medical domains [Paper] [Code]
• [arXiv] BiMediX: Bilingual medical mixture of experts LLM [Paper] [Code]
• [arXiv] OncoGPT: A medical conversational model tailored with oncology domain expertise on a large language model Meta-AI (LLaMA) [Paper] [Code]
• [arXiv] JMLR: Joint medical LLM and retrieval training for enhancing reasoning and professional question answering capability [Paper]

2023

• [Bioinformatics] MedCPT: A method for zero-shot biomedical information retrieval using contrastive learning with PubMedBERT. [Paper] [Code]
• [arXiv] Pmc-llama: Towards building open-source language models for medicine. [Paper] [Code]
• [arXiv] Meditron-70b: Scaling medical pretraining for large language models. [Paper] [Code]
• [arXiv] Qilin-med: Multi-stage knowledge injection advanced medical large language model. [Paper] [Code]
• [arXiv] Huatuogpt-ii, one-stage training for medical adaption of llms. [Paper] [Code]
• [NPJ Digit. Med.] A study of generative large language model for medical research and healthcare. [Paper] [Code]
• [arXiv] From beginner to expert: Modeling medical knowledge into general llms. [Paper]
• [arXiv] Huatuo: Tuning llama model with chinese medical knowledge. [Paper] [Code]
• [arXiv] Chatdoctor: A medical chat model fine-tuned on a large language model meta-ai (llama) using medical domain knowledge. [Paper] [Code]
• [arXiv] Medalpaca–an open-source collection of medical conversational ai models and training data. [Paper] [Code]
• [arXiv] Alpacare: Instruction-tuned large language models for medical application. [Paper] [Code]
• [arXiv] Huatuogpt, towards taming language model to be a doctor. [Paper] [Code]
• [arXiv] Doctorglm: Fine-tuning your chinese doctor is not a herculean task. [Paper] [Code]
• [arXiv] Bianque: Balancing the questioning and suggestion ability of health llms with multi-turn health conversations polished by chatgpt. [Paper] [Code]
• [arXiv] Taiyi: A bilingual fine-tuned large language model for diverse biomedical tasks. [Paper] [Code]
• [Github] Visual med-alpaca: A parameter-efficient biomedical llm with visual capabilities. [Code]
• [arXiv] Ophglm: Training an ophthalmology large languageand-vision assistant based on instructions and dialogue. [Paper] [Code]
• [arXiv] Chatcad: Interactive computer-aided diagnosis on medical image using large language models. [Paper] [Code]
• [arXiv] Chatcad+: Towards a universal and reliable interactive cad using llms. [Paper] [Code]
• [arXiv] Deid-gpt: Zero-shot medical text de-identification by gpt-4. [Paper] [Code]
• [arXiv] Can generalist foundation models outcompete special-purpose tuning? case study in medicine. [Paper] [Code]
• [arXiv] Medagents: Large language models as collaborators for zero-shot medical reasoning. [Paper] [Code]
• [AIME] Soft-prompt tuning to predict lung cancer using primary care free-text dutch medical notes. [Paper] [Code]
• [arXiv] Clinical decision transformer: Intended treatment recommendation through goal prompting. [Paper] [Code]
• [Nature] Large language models encode clinical knowledge [Paper]
• [arXiv] Towards expert-level medical question answering with large language models [Paper]
• [arXiv] Gpt-doctor: Customizing large language models for medical consultation [Paper]
• [arXiv] Clinicalgpt: Large language models finetuned with diverse medical data and comprehensive evaluation [Paper]
• [arXiv] Leveraging a medical knowledge graph into large language models for diagnosis prediction [Paper]

2022

• [NPJ Digit. Med.] A large language model for electronic health records. [Paper] [Code]
• [AMIA Annu. Symp. Proc.] Healthprompt: A zero-shot learning paradigm for clinical natural language processing. [Paper]
• [BioNLP] Position-based prompting for health outcome generation [Paper]

2021

• [ACM Trans. Comput. Healthc.] Domain-specific language model pretraining for biomedical natural language processing. [Paper] [Code]

2020

• [JMIR Med. Info.] Modified bidirectional encoder representations from transformers extractive summarization model for hospital information systems based on character-level tokens (alphabert): development and performance evaluation. [Paper] [Code]
• [Scientific reports] Behrt: transformer for electronic health records. [Paper] [Code]
• [BioNLP] BioBART: Pretraining and evaluation of a biomedical generative language model. [Paper] [Code]

2019

• [NPJ Digit. Med.] ClinicalBERT: A hybrid learning model for natural language inference in healthcare using BERT. [Paper] [Code]
• [Method. Biochem. Anal.] Biobert: a pre-trained biomedical language representation model for biomedical text mining. [Paper] [Code]

VFM methods

2024

• [arXiv] USFM: A universal ultrasound foundation model generalized to tasks and organs towards label efficient image analysis. [paper]
• [CVPR] VoCo: A simple-yet-effective volume contrastive learning framework for 3D medical image analysis. [paper][Code]
• [NeurIPS] LVM-Med: Learning large-scale self-supervised vision models for medical imaging via second-order graph matching. [paper] [Code]
• [Nature Medicine] Towards a general-purpose foundation model for computational pathology. [paper] [Code]
• [arXiv] RudolfV: A foundation model by pathologists for pathologists. [paper] [Code]
• [Nature Communications] Segment anything in medical images. [paper] [Code]
• [ICASSP] SAM-OCTA: A fine-tuning strategy for applying foundation model to OCTA image segmentation tasks.[paper] [Code]
• [WACV] AFTer-SAM: Adapting SAM with axial fusion transformer for medical imaging segmentation. [paper]
• [MIDL] Adaptivesam: Towards efficient tuning of sam for surgical scene segmentation. [paper] [Code]
• [arXiv] SegmentAnyBone: A universal model that segments any bone at any location on MRI [paper] [Code]
• [SSRN] Swinsam: Fine-grained polyp segmentation in colonoscopy images via segment anything model integrated with a Swin transformer decoder. [paper]
• [AAAI] Surgicalsam: Efficient class promptable surgical instrument segmentation [paper] [Code]
• [Medical Image Analysis] Prompt tuning for parameter-efficient medical image segmentation. [paper] [Code]

2023

• [ICCV] UniverSeg: Universal medical image segmentation. [paper] [Code]
• [arXiv] STU-Net: Scalable and transferable medical image segmentation models empowered by large-scale supervised pre-training. [paper] [Code]
• [arXiv] SAM-Med3D. [paper] [Code]
• [Nature] A foundation model for generalizable disease detection from retinal images. [paper]
• [arXiv] VisionFM: a multi-modal multi-task vision foundation model for generalist ophthalmic Artificial Intelligence. [paper]
• [arXiv] Segvol: Universal and interactive volumetric medical image segmentation. [paper] [Code]
• [MICCAI] Models Genesis: Generic autodidactic models for 3D medical image analysis. [paper] [Code]
• [MICCAI] Deblurring masked autoencoder is better recipe for ultrasound image recognition. [paper] [Code]
• [arXiv] Mis-fm: 3d medical image segmentation using foundation models pretrained on a large-scale unannotated dataset. [paper] [Code]
• [MICCAI] Foundation model for endoscopy video analysis via large-scale self-supervised pre-train. [paper][Code]
• [MIDL] MoCo pretraining improves representation and transferability of chest X-ray models. [paper] [Code]
• [arXiv] BROW: Better features for whole slide image based on self-distillation[paper]
• [arXiv] Computational pathology at health system scale--self-supervised foundation models from three billion images. [paper]
• [CVPR] Geometric visual similarity learning in 3D medical image self-supervised pre-training.[paper] [Code]
• [arXiv] Virchow: A million-slide digital pathology foundation model.[paper] [Code]
• [arXiv] Ma-sam: Modality-agnostic sam adaptation for 3d medical image segmentation.[paper] [Code]
• [ICCV] Comprehensive multimodal segmentation in medical imaging: combining YOLOv8 with SAM and HQ-SAM models. [paper]
• [arXiv] 3DSAM-adapter: Holistic adaptation of SAM from 2D to 3D for promptable medical image segmentation.[paper] [Code]
• [arXiv] Part to whole: Collaborative prompting for surgical instrument segmentation. [paper] [Code]
• [arXiv] Towards general purpose vision foundation models for medical image analysis: An experimental study of DINOv2 on radiology benchmarks.[paper] [Code]
• [arXiv] Skinsam: Empowering skin cancer segmentation with segment anything model.[paper]
• [arXiv] Polyp-sam: Transfer sam for polyp segmentation. [paper] [Code]
• [arXiv] Customized segment anything model for medical image segmentation. [paper] [Code]
• [arXiv] Ladder fine-tuning approach for SAM integrating complementary network. [paper] [Code]
• [arXiv] Cheap lunch for medical image segmentation by fine-tuning sam on few exemplars. [paper]
• [arXiv] SemiSAM: Exploring SAM for enhancing semi-supervised medical image segmentation with extremely limited annotations. [paper]
• [IWMLMI] Mammo-sam: Adapting foundation segment anything model for automatic breast mass segmentation in whole mammograms. [paper]
• [arXiv] Promise: Prompt-driven 3D medical image segmentation using pretrained image foundation models. [paper] [Code]
• [arXiv] Medical sam adapter: Adapting segment anything model for medical image segmentation. [paper] [Code]
• [arXiv] SAM-Med2D [paper] [Code]
• [arXiv] Medivista-sam: Zero-shot medical video analysis with spatio-temporal sam adaptation. [paper] [Code]
• [arXiv] Samus: Adapting segment anything model for clinically-friendly and generalizable ultrasound image segmentation. [paper]
• [MICCAI] Input augmentation with sam: Boosting medical image segmentation with segmentation foundation model. [paper] [Code]
• [arXiv] AutoSAM: Adapting SAM to medical images by overloading the prompt encoder. [paper]
• [arXiv] DeSAM: Decoupling segment anything model for generalizable medical image segmentation [paper] [Code]
• [bioRxiv] A foundation model for cell segmentation.[paper] [Code]
• [MICCAI] SAM-U: Multi-box prompts triggered uncertainty estimation for reliable SAM in medical image. [paper]
• [MICCAI] Sam-path: A segment anything model for semantic segmentation in digital pathology. [paper]
• [arXiv] All-in-sam: from weak annotation to pixel-wise nuclei segmentation with prompt-based finetuning.[paper]
• [arXiv] Polyp-sam++: Can a text guided sam perform better for polyp segmentation? [paper] [Code]
• [arXiv] Segment anything model with uncertainty rectification for auto-prompting medical image segmentation. [paper]
• [arXiv] MedLSAM: Localize and segment anything model for 3D medical images. [paper] [Code]
• [arXiv] nnSAM: Plug-and-play segment anything model improves nnUNet performance. [paper] [Code]
• [arXiv] EviPrompt: A training-free evidential prompt generation method for segment anything model in medical images. [paper]
• [arXiv] One-shot localization and segmentation of medical images with foundation models. [paper]
• [arXiv] Samm (segment any medical model): A 3d slicer integration to sam. [paper] [Code]
• [arXiv] Task-driven prompt evolution for foundation models.[paper]

2022

• [Machine Learning with Applications] Self supervised contrastive learning for digital histopathology. [paper] [Code]
• [Medical Image Analysis] Transformer-based unsupervised contrastive learning for histopathological image classification. [paper] [Code]
• [arXiv] Self-supervised learning from 100 million medical images. [paper]
• [CVPR] Self-supervised pre-training of swin transformers for 3d medical image analysis.[paper] [Code]

2021

• [Medical Image Analysis] Models genesis. [paper] [Code]
• [Medical Imaging with Deep Learning] MoCo pretraining improves representation and transferability of chest X-ray models. [paper]
• [IEEE transactions on medical imaging] Transferable visual words: Exploiting the semantics of anatomical patterns for self-supervised learning.[Paper]

2020

• [MICCAI] Comparing to learn: Surpassing imageNet pretraining on radiographs by comparing image representations. [paper] [Code]

2019

• [arXiv] Med3D: Transfer learning for 3D medical image analysis. [paper] [Code]

BFM methods

2024

• [Nucleic Acids Research] Multiple sequence alignment-based RNA language model and its application to structural inference. [Paper], [Code]
• [Nature Methods] scGPT: toward building a foundation model for single-cell multi-omics using generative AI. [Paper], [Code]
• [Nature Machine Intelligence] A 5’ UTR language model for decoding untranslated regions of mRNA and function predictions. [Paper], [Code]
• [ICLR 2024] CellPLM: Pre-training of Cell Language Model Beyond Single Cells. [Paper], [Code] 2023
• [arXiv] DNAGPT: A generalized pre-trained tool for versatile DNA sequence analysis tasks. [Paper], [Code]
• [arXiv] HyenaDNA: Long-range genomic sequence modeling at single nucleotide resolution. [Paper], [Code]
• [Nature Biotechnology] Large language models generate functional protein sequences across diverse families. [Paper], [Code]
• [Cell Systems] ProGen2: Exploring the boundaries of protein language models. [Paper], [Code]
• [Nature] Transfer learning enables predictions in network biology. [Paper], [Code]
• [arXiv] DNABERT-2: Efficient foundation model and benchmark for multi-species genome. [Paper], [Code]
• [bioRxiv] The nucleotide transformer: Building and evaluating robust foundation models for human genomics. [Paper], [Code]
• [bioRxiv] GENA-LM: A family of open-source foundational models for long DNA sequences. [Paper], [Code]
• [bioRxiv] Self-supervised learning on millions of pre-mRNA sequences improves sequence-based RNA splicing prediction. [Paper], [Code]
• [bioRxiv] Deciphering 3’ UTR mediated gene regulation using interpretable deep representation learning. [Paper], [Code]
• [Science] Evolutionary-scale prediction of atomic-level protein structure with a language model. [Paper], [Code]
• [bioRxiv] Universal cell embeddings: A foundation model for cell biology. [Paper], [Code]
• [bioRxiv] Large scale foundation model on single-cell transcriptomics. [Paper], [Code]
• [arXiv] Large-scale cell representation learning via divide-and-conquer contrastive learning. [Paper], [Code]
• [bioRxiv] CodonBERT: Large language models for mRNA design and optimization. [Paper], [Code]
• [bioRxiv] xTrimoPGLM: Unified 100B-scale pre-trained transformer for deciphering the language of protein. [Paper]
• [bioRxiv] GenePT: A simple but effective foundation model for genes and cells built from ChatGPT. [Paper], [Code]
• [bioRxiv] scELMo: Embeddings from language models are good learners for single-cell data analysis. [Paper], [Code]
• [bioRxiv] Evaluating the Utilities of Foundation Models in Single-cell Data Analysis. [Paper], [Code]
• [bioRxiv] GeneCompass: Deciphering Universal Gene Regulatory Mechanisms with Knowledge-Informed Cross-Species Foundation Model. [Paper], [Code]

2022

• [Nature Machine Intelligence] scBERT as a large-scale pretrained deep language model for cell type annotation of single-cell RNA-seq data. [Paper], [Code]
• [bioRxiv] Interpretable RNA foundation model from unannotated data for highly accurate RNA structure and function predictions. [Paper], [Code]
• [NAR Genomics & Bioinformatics] Informative RNA base embedding for RNA structural alignment and clustering by deep representation learning. [Paper], [Code]
• [Nature Biotechnology] Single-sequence protein structure prediction using language models and deep learning. [Paper], [Code]

2021

• [Bioinformatics] DNABERT: pre-trained Bidirectional Encoder Representations from Transformers model for DNA-language in genome. [Paper], [Code]
• [IEEE TPAMI] ProtTrans: Toward understanding the language of life through self-supervised learning. [Paper], [Code]
• [ICML 2021] MSA Transformer. [Paper], [Code]
• [PNAS] Biological structure and function emerge from scaling unsupervised learning to 250 million protein sequences. [Paper], [Code]
• [Nature] Highly accurate protein structure prediction with AlphaFold. [Paper], [Code]
• [arXiv] Multi-modal self-supervised pre-training for regulatory genome across cell types. [Paper], [Code]

MFM methods

2024

• [ICASSP] Etp: Learning transferable ecg representations via ecg-text pretraining. [Paper]
• [NeurIPS] Med-unic: Unifying cross-lingual medical vision language pre-training by diminishing bias. [Paper] [Code]
• [NeurIPS] Quilt-1m: One million image-text pairs for histopathology. [Paper] [Code]
• [Nature Medicine] A visual-language foundation model for computational pathology. [Paper]
• [NeurIPS] LLaVA-Med: Training a large language-and-vision assistant for biomedicine in one day. [Paper] [Code]
• [AAAI] PathAsst: Generative foundation AI assistant for pathology. [Paper] [Code]
• [WACV] I-AI: A controllable & interpretable AI system for decoding radiologists’ intense focus for accurate CXR diagnoses. [Paper] [Code]
• [arXiv] M3D: Advancing 3D medical image analysis with multi-modal large language models. [Paper] [Code]

2023

• [ICLR] Advancing radiograph representation learning with masked record modeling. [Paper] [Code]
• [arXiv] BiomedGPT: A unified and generalist biomedical generative pre-trained transformer for vision, language, and multimodal Tasks. [Paper] [Code]
• [arXiv] BiomedCLIP: a multimodal biomedical foundation model pretrained from fifteen million scientific image-text pairs. [Paper] [Code]
• [arXiv] Towards generalist foundation model for radiology by leveraging web-scale 2D&3D medical data. [Paper] [Code]
• [CVPR] Visual language pretrained multiple instance zero-shot transfer for histopathology images. [Paper] [Code]
• [ICCV] Medklip: Medical knowledge enhanced language-image pre-training. [Paper] [Code]
• [arXiv] UniBrain: Universal brain MRI diagnosis with hierarchical knowledge-enhanced pre-training. [Paper] [Code]
• [EACL] PubMedCLIP: How much does CLIP benefit visual question answering in the medical domain. [Paper] [Code]
• [MICCAI] M-FLAG: Medical vision-language pre-training with frozen language models and latent space geometry optimization. [Paper] [Code]
• [arXiv] IMITATE: Clinical prior guided hierarchical vision-language pre-training. [Paper]
• [arXiv] CXR-CLIP: Toward large scale chest X-ray language-image pre-training. [Paper] [Code]
• [BIBM] UMCL: Unified medical image-text-label contrastive learning with continuous prompt. [Paper]
• [Nature Communications] Knowledge-enhanced visual-language pre-training on chest radiology images. [Paper]
• [Nature Machine Intelligence] Multi-modal molecule structure–text model for text-based retrieval and editing. [Paper] [Code]
• [MICCAI] Clip-lung: Textual knowledge-guided lung nodule malignancy prediction. [Paper]
• [MICCAI] Pmc-clip: Contrastive language-image pre-training using biomedical documents. [Paper] [Code]
• [arXiv] Enhancing representation in radiography-reports foundation model: A granular alignment algorithm using masked contrastive learning. [Paper] [Code]
• [ICCV] Prior: Prototype representation joint learning from medical images and reports. [Paper] [Code]
• [MICCAI] Masked vision and language pre-training with unimodal and multimodal contrastive losses for medical visual question answering. [Paper] [Code]
• [arXiv] T3d: Towards 3d medical image understanding through vision-language pre-training. [Paper]
• [MICCAI] Gene-induced multimodal pre-training for imageomic classification. [Paper] [Code]
• [arXiv] A text-guided protein design framework. [Paper] [Code]
• [Nature Medicine] A visual--language foundation model for pathology image analysis using medical Twitter. [Paper] [Code]
• [arXiv] Towards generalist biomedical ai. [Paper] [Code]
• [ML4H] Med-Flamingo: A multimodal medical few-shot learner. [Paper] [Code]
• [MLMIW] Exploring the transfer learning capabilities of CLIP on domain generalization for diabetic retinopathy. [Paper] [Code]
• [MICCAI] Open-ended medical visual question answering through prefix tuning of language models. [Paper] [Code]
• [arXiv] Qilin-Med-VL: Towards chinese large vision-language model for general healthcare. [Paper] [Code]
• [arXiv] A foundational multimodal vision language AI assistant for human pathology. [Paper]
• [arXiv] Effectively fine-tune to improve large multimodal models for radiology report generation. [Paper]
• [MLMIW] Multi-modal adapter for medical vision-and-language learning. [Paper]
• [arXiv] Text-guided foundation model adaptation for pathological image classification. [Paper] [Code]
• [arXiv] XrayGPT: Chest radiographs summarization using medical vision-language models. [Paper] [Code]
• [MICCAI] Xplainer: From X-Ray observations to explainable zero-shot diagnosis. [Paper] [Code]
• [MICCAI] Multiple prompt fusion for zero-shot lesion detection using vision-language models. [Paper]

2022

• [JMLR] Contrastive learning of medical visual representations from paired images and text. [Paper] [Code]
• [ECCV] Joint learning of localized representations from medical images and reports. [Paper]
• [NeurIPS] Multi-granularity cross-modal alignment for generalized medical visual representation learning. [Paper] [Code]
• [AAAI] Clinical-BERT: Vision-language pre-training for radiograph diagnosis and reports generation. [Paper]
• [MICCAI] Multi-modal masked autoencoders for medical vision-and-language pre-training. [Paper] [Code]
• [JBHI] Multi-modal understanding and generation for medical images and text via vision-language pre-training. [Paper] [Code]
• [ACM MM] Align, reason and learn: Enhancing medical vision-and-language pre-training with knowledge. [Paper] [Code]
• [ECCV] Making the most of text semantics to improve biomedical vision–language processing. [Paper]
• [Nature Biomedical Engineering] Expert-level detection of pathologies from unannotated chest x-ray images via self-supervised learning. [Paper] [Code]
• [arXiv] RoentGen: Vision-language foundation model for chest X-ray generation. [Paper]
• [arXiv] Adapting pretrained vision-language foundational models to medical imaging domains. [Paper]
• [arXiv] Medical image understanding with pretrained vision language models: A comprehensive study. [Paper]
• [EMNLP] Medclip: Contrastive learning from unpaired medical images and text. [Paper] [Code]
• [MICCAI] Breaking with fixed set pathology recognition through report-guided contrastive training. [Paper]

2021

• [arXiv] MMBERT: Multimodal BERT pretraining for improved medical VQA. [Paper] [Code]
• [ICCV] GLoRIA: A multimodal global-local representation learning framework for label-efficient medical image recognition. [Paper] [Code]

Datasets

LFM datasets

VFM datasets

BFM datasets

MFM datasets

Large-scale comprehensive databases

Other resources

Lectures and tutorials

• Bioinformatics - Geneformer Tutorial (unofficial)

Blogs

• How Foundation Models Can Advance AI in Healthcare
• The Shaky Foundations of Foundation Models in Healthcare

Related awesome repositories

• Awesome-Healthcare-Foundation-Models
• Awesome-Foundation-Models