Domain Generalization and Foundation Model in Medical Image Analysis

🔥 This is a repository for organizing papers ,codes, and etc related to Domain Generalization and Foundation model in Medical Image Analysis (DG&FM in MedIA).

💗 Medical Image Analysis (MedIA) plays a critical role in computer aided diagnosis system, enabling accurate diagnosis and assessment for various diseases. Over the last decade, deep learning (DL) has demonstrated great success in automating various MedIA tasks such as disease diagnosis, lesion segmentation, prognosis prediction, etc. Despite their success, in many real-world healthcare scenarios, the difference in the image acquisition, such as device manufacturer, scanning protocol, image sequence, and modality, introduces domain shifts, resulting in a significant decline in performance when deploying the well-trained model to clinical sites with different data distributions. Therefore, enhancing the generalization ability of DL models in MedIA is crucial in both clinical and academic fields. Domain generalization (DG), as an effective method to improve the generalization performance of task-specific models, can effectively mitigate the performance degradation caused by domain shifts in medical images, such as cross-center, cross-sequence, and cross-modality variations. Recently, with the explosive growth of data and advancements in computational resources, Foundation Model (FM) has addressed the domain shift issue in a more direct manner by collecting a vast amount of diverse data, effectively preventing domain shifts at the source. It can handle a wide variety of tasks, including entirely new tasks that it has never encountered before. However, compared to task-specific DG models, FM offers increased task diversity and flexibility. Nonetheless, challenges such as medical data privacy concerns, data-sharing restrictions, the need for manual annotations by medical experts, and deployment demands persist. Therefore, we maintain that both DG and FM have their own merits and continue to hold significant research value.

🎯 We hope that this repository can provide assistance to researchers and practitioners in medical image analysis, domain generalization and foundation models.

🚀 New Updates:

• 06/01/2025 : We have modified the presentation format to a table, which makes it easier for readers to review.
• 25/12/2024 : We have added a Universal Segmentation Foundational Model branch.
• 08/02/2024 : We released this repo for organizing papers ,codes, and etc related to domain generalization for medical image analysis.

Table of Contents

• Domain Generalization and Foundation Model in Medical Image Analysis
• Table of Contents
• Papers on Domian Generalization (ongoing)
  • Data Manipulation Level
    • Data Augmentation
      • Normalization-based
      • Randomization-based
      • Adversarial-based
    • Data Generation
  • Feature Level Generalization
    • Invariant Feature Representation
      • Feature normalization
      • Explicit feature alignment
      • Domain adversarial learning
    • Feature disentanglement
      • Multi-component learning
      • Generative Learning
  • Model Training Level
    • Learning Strategy
      • Ensemble Learning
      • Meta Learning
      • Self-supervised Learning
    • Optimization Strategy
  • Model Test Level
    • Test-time Adaptation
• Papers on Universal Foundation Model
  • Survey
  • Visual Foundation Models
    • Interactive
    • Few-shot/One-shot
  • Multimodal Foundation Models
    • Contrastive
    • Generative
    • Conversational
• Datasets
• Libraries
• Other Resources
• Contact
• Acknowledgements
• Contributors

Papers on Domian Generalization (ongoing)

Data Manipulation Level

Data Augmentation

Augmentation is widely employed in vision tasks to mitigate overfitting and improve generalization capacity, including operations like flipping, cropping, color jittering, noise addition, and others. For domain generalization in medical image analysis, augmentation methods can be broadly categorized as randomization-based, adversarial-based, and normalization-based.

Normalization-based

Normalization-based methods aims to normalize the raw intensity values or statistics to reduce the impact of variations in image intensity across different domains. Specifically, these methods are usually employed for specific tasks, such as pathological images testtest.

Randomization-based

The goal of randomization-based methods is to generate novel input data by applying random transformations to the image-space, frequency-space and feature space.

Image-space

Frequency-space

Feature-space

Adversarial-based

Adversarial-based data augmentation methods are driven by adversarial training, aiming to maximize the diversity of data while simultaneously constraining its reliability.

Data Generation

Data generation is devoted to utilizing generative models such as Variational Autoencoder (VAE), Generative Adversarial Networks (GANs), Diffusion Models and etc., to generate fictional and novel samples. With source domain data becoming more complex, diverse, and informative, the generalization ability can be increased.

Feature Level Generalization

Invariant Feature Representation

For medical image analysis, a well-generalized model focuses more on task-related semantic features while disregarding task-unrelated style features. In this regard, three types of methods have been extensively investigated: feature normalization, explicit feature alignment, and domain adversarial learning.

Feature normalization

This line of methods aim to enhance the generalization ability of models by centering, scaling, decorrelating, standardizing, and whitening extracted feature distributions. This process aids in accelerating the convergence of algorithms and prevents features with larger scales from overpowering those with smaller ones. Common techniques include traditional scaling methods like min-max and z-score normalization, as well as deep learning methods such as batch, layer, and instance normalization.

Explicit feature alignment

Explicit feature alignment methods attempt to remove domain shifts by reducing the discrepancies in feature distributions across multiple source domains, thereby facilitating the learning of domain-invariant feature representations.

Domain adversarial learning

Domain-adversarial training methods are widely used for learning domain-invariant representations by introducing a domain discriminator in an adversarial relationship with the feature extractor

Feature disentanglement

Feature disentanglement methods aim to decompose the features of input samples into domain-invariant (task-unrelated) and domain-specific (task-related) components, i.e., $\mathbf{z} = [\mathbf{z}\text{invariant}, \mathbf{z}\text{specific}] \in \mathcal{Z}$. The objective of robust generalization models is to concentrate exclusively on the task-related feature components $\mathbf{z}\text{invariant}$ while disregarding the task-unrelated ones $\mathbf{z}\text{specific}$. The mainstream methods of feature disentanglement mainly include multi-component learning and generative modeling.

Multi-component learning

Multi-component learning achieves feature disentanglement by designing different components to separately extract domain-invariant features and domain-specific features, thereby achieving feature decoupling.

Generative Learning

Generative models are also effective techniques for traditional feature disentanglement, such as InfoGAN and $\beta$-VAE. For domain generalization, generative learning based disentanglement methods attempt to elucidate the sample generation mechanisms from the perspectives of domain, sample, and label, thereby achieving feature decomposition.

Model Training Level

Learning Strategy

Learning strategies have gained significant attention in tackling domain generalization challenges across various fields. They leverage generic learning paradigms to improve model generalization performance, which can be mainly categorized into three categories: ensemble learning, meta-learning, and self-supervised learning.

Ensemble Learning

Ensemble learning is a machine learning technique where multiple models are trained to solve the same problem. For domain generalization, different models can capture domain-specific patterns and representations, so their combination could lead to more robust predictions.

Meta Learning

Meta-learning, also known as learning to learn, is a machine learning method focused on designing algorithms that can generalize knowledge from diverse tasks. In medical domain generalization tasks, it plays a significant role in addressing the challenge of expensive data collecting and annotating, which divide the source domain(s) into meta-train and meta-test sets to simulate domain shift.

Self-supervised Learning

Self-supervised learning is a machine learning method where a model learns general representations from input data without explicit supervision. These representations enhance the model's generalization capability, enabling it to mitigate domain-specific biases. This approach is particularly valuable in scenarios where labeled data is scarce or costly to obtain and annotate, such as in medical imaging.

Optimization Strategy

Optimization strategies play a crucial role in minimizing overfitting to specific domains, which is achieved by adjusting hyperparameters, selecting appropriate loss functions, regularization techniques, and optimization algorithms.

Model Test Level

Test-time Adaptation

Papers on Universal Foundation Model

Medical image segmentation tasks encompass diverse imaging modalities, such as magnetic resonance imaging (MRI), X-ray, computed tomography (CT), and microscopy; various biomedical domains, including the abdomen, chest, brain, retina, and individual cells; and multiple label types within a region, such as heart valves or chambers. Traditional task-specific models are designed to train and test on a single, specific dataset. In contrast, universal foundation models aim to learn a single, generalizable medical image segmentation model capable of performing well across a wide range of tasks, including those significantly different from those encountered during training, without requiring retraining.

Survey

Visual Foundation Models

Interactive

Interactive segmentation paradigm means the foundation model segments the target following the user-given prompts, such as a point, a bounding box (BBox), doodles or free text-like descriptions.

Few-shot/One-shot

In few-shot/one-shot setting, a pre-trained foundationa model needs one or few labeled samples as the ’supportive examples’, to grasp a new specific task.

Multimodal Foundation Models

Contrastive

Contrastive textually prompted models are increasingly recognized as foundational models for medical imaging. They learn representations that capture the semantics and relationships between medical images and their corresponding textual prompts. By leveraging contrastive learning objectives, these models bring similar image-text pairs closer in the feature space while pushing dissimilar pairs apart.

Generative

Generative models represent another category within textually prompted models for medical imaging. These models are designed to generate realistic medical images based on textual prompts or descriptions. They utilize techniques such as variational autoencoders (VAEs) and generative adversarial networks (GANs) to learn the underlying distribution of medical images, enabling the creation of new samples that align with the provided prompts.

Conversational

Conversational textually prompted models are designed to enable interactive dialogues between medical professionals and the model by fine-tuning foundational models on specific instruction sets. These models enhance communication and collaboration, allowing medical experts to ask questions, provide instructions, and seek explanations related to medical images.

Datasets

We list the widely used benchmark datasets for domain generalization including classification and segmentation.

Libraries

We list the libraries of domain generalization.

• Transfer Learning Library (thuml) for Domain Adaptation, Task Adaptation, and Domain Generalization.
• DomainBed (facebookresearch) is a suite to test domain generalization algorithms.
• DeepDG (Jindong Wang): Deep domain generalization toolkit, which is easier then DomainBed.
• Dassl (Kaiyang Zhou): A PyTorch toolbox for domain adaptation, domain generalization, and semi-supervised learning.
• TorchSSL (Jindong Wang): A open library for semi-supervised learning.

Other Resources

• A collection of domain generalization papers organized by amber0309.
• A collection of domain generalization papers organized by jindongwang.
• A collection of papers on domain generalization, domain adaptation, causality, robustness, prompt, optimization, generative model, etc, organized by yfzhang114.
• A collection of awesome things about domain generalization organized by junkunyuan.

Contact

• If you would like to add/update the latest publications / datasets / libraries, please directly add them to this README.md.
• If you would like to correct mistakes/provide advice, please contact us by email (nzw@zju.edu.cn).
• You are welcomed to update anything helpful.

Acknowledgements

• We refer to Generalizing to Unseen Domains: A Survey on Domain Generalization to design the hierarchy of the Contents.
• We refer to junkunyuan, amber0309, and yfzhang114 to design the details of the papers and datasets.

Contributors