Balanced Multimodal Learning

A curated list of balanced multimodal learning methods.

News

• You are also welcome to join our Slack workspace: https://join.slack.com/t/balancedmulti-mhs9373/shared_invite/zt-3aj46comq-CNWrkqqGI9Z2kNn64aU9FQ

Table of contents

• Balanced Multimodal Learning
• News
• Table of contents
  • GeWu-Lab paper
  • Other balanced multimodal learning methods
  • Methodology Classification (Refer to the paper BalanceBenchmark)
    • Data
    • Feed-forward
    • Objective
    • Optimization
  • BML in MLLMs

GeWu-Lab paper

[CVPR-2022]
Balanced Multimodal Learning via On-the-fly Gradient Modulation
Code

[ICASSP-2023]
MMCosine: Multi-Modal Cosine Loss Towards Balanced Audio-Visual Fine-Grained Learning
Code

[ICLR-2024]
Quantifying and Enhancing Multi-modal Robustness with Modality Preference
Code

[CVPR-2024]
Enhancing Multimodal Cooperation via Sample-level Modality Valuation
Code

[ICML-2024]
MMPareto: Boosting Multimodal Learning with Innocent Unimodal Assistance
Code

[ECCV-2024]
Diagnosing and Re-learning for Balanced Multimodal Learning
Code

[CVPR-2025]
Adaptive Unimodal Regulation for Balanced Multimodal Information Acquisition
Code

Other balanced multimodal learning methods

• What Makes Training Multi-Modal Classification Networks Hard? [CVPR 2020]
• Learning to Balance the Learning Rates between Various Modalities via Adaptive Tracking Factor [IEEE Signal Processing Letters 2021]
• Audiovisual SlowFast Networks for Video Recognition [arXiv 2020]
• Joint Audio-Visual Deepfake Detection [ICCV 2021]
• Delving into Deep Imbalanced Regression [ICML2021]
• Characterizing and Overcoming the Greedy Nature of Learning in Multi-modal Deep Neural Networks [ICML 2022]
• MCL: A Contrastive Learning Method for Multimodal Data Fusion in Violence Detection [IEEE Signal Processing Letters 2022]
• Balanced MSE for Imbalanced Visual Regression[CVPR2022]
• On Uni-Modal Feature Learning in Supervised Multi-Modal Learning [ICML 2023]
• Calibrating Multimodal Learning [ICML 2023]
• Multimodal Pre-Training with Self-Distillation for Product Understanding in E-Commerce [WSDM 2023]
• PMR: Prototypical Modal Rebalance for Multimodal Learning [CVPR 2023]
• Graph Interactive Network with Adaptive Gradient for Multi-Modal Rumor Detection [ICMR 2023]
• Multimodal Imbalance-Aware Gradient Modulation for Weakly-supervised Audio-Visual Video Parsing [IEEE Transactions on Circuits and Systems for Video Technology 2023]
• Multimodal Temporal Attention in Sentiment Analysis [MuSe 2023]
• Utilizing Greedy Nature for Multimodal Conditional Image Synthesis in Transformers [TMM 2023]
• Variational Probabilistic Fusion Network for RGB-T Semantic Segmentation [arXiv 2023]
• Boosting Multi-modal Model Performance with Adaptive Gradient Modulation [ICCV 2023]
• Adaptive Mask Co-Optimization for Modal Dependence in Multimodal Learning [ICASSP 2024]
• Improving Multimodal Learning with Multi-Loss Gradient Modulation [arXiv 2024]
• Learning to Rebalance Multi-Modal Optimization by Adaptively Masking Subnetworks [arXiv 2024]
• Suppress and Rebalance: Towards Generalized Multi-Modal Face Anti-Spoofing [CVPR 2024]
• C2KD: Bridging the Modality Gap for Cross-Modal Knowledge Distillation [CVPR 2024]
• Multimodal Representation Learning by Alternating Unimodal Adaptation [CVPR 2024]
• Balanced Multi-modal Federated Learning via Cross-Modal Infiltration [arXiv 2024]
• Balancing Multimodal Learning via Online Logit Modulation [IJCAI 2024]
• Enhancing Unimodal Features Matters: A Multimodal Framework for Building Extraction [TGRS 2024]
• Suppress and Rebalance: Towards Generalized Multi-Modal Face Anti-Spoofing [CVPR 2024]
• Understanding Unimodal Bias in Multimodal Deep Linear Networks [ICML 2024]
• Facilitating Multimodal Classification via Dynamically Learning Modality Gap [NIPS 2024]
• Ada2I: Enhancing Modality Balance for Multimodal Conversational Emotion Recognition [ACM MM 2024]
• Modality-Balanced Learning for Multimedia Recommendation [ACM MM 2024]
• ReconBoost: Boosting Can Achieve Modality Reconcilement. [ICML 2024]
• Classifier-guided Gradient Modulation for Enhanced Multimodal Learning [arXiv 2024]
• DynCIM: Dynamic Curriculum for Imbalanced Multimodal Learning [arXiv 2025]
• Rethinking Multimodal Learning from the Perspective of Mitigating Classification Ability Disproportion [arXiv 2025]
• Asymmetric Reinforcing Against Multi-Modal Representation Bias [AAAI 2025]
• Spiking Neural Networks with Temporal Attention-Guided Adaptive Fusion for imbalanced Multi-modal Learning [arXiv 2025]
• Reliable and Balanced Transfer Learning for Generalized Multimodal Face Anti-Spoofing [TPAMI 2025]
• Shaping Initial State Prevents Modality Competition in Multi-modal Fusion: A Two-stage Scheduling Framework via Fast Partial Information Decomposition [arXiv 2025]
• AIM: Adaptive Intra-Network Modulation for Balanced Multimodal Learning [arXiv 2025]
• Balanced Multimodal Learning: An Unidirectional Dynamic Interaction Perspective [arXiv 2025]
• DMAF-Net: An Effective Modality Rebalancing Framework for Incomplete Multi-Modal Medical Image Segmentation [arXiv 2025]
• Towards Explainable Fusion and Balanced Learning in Multimodal Sentiment Analysis [arXiv 2025]
• Learning Optimal Multimodal Information Bottleneck Representations [arXiv 2025]
• Modality Equilibrium Matters: Minor-Modality-Aware Adaptive Alternating for Cross-Modal Memory Enhancement [arXiv 2025]
• MIDAS: Misalignment-based Data Augmentation Strategy for Imbalanced Multimodal Learning [NeurIPS 2025]

Methodology Classification (Refer to the paper BalanceBenchmark)

Data

---

• Enhancing Multimodal Cooperation via Sample-level Modality Valuation [CVPR 2024]

Feed-forward

---

• Multimodal Representation Learning by Alternating Unimodal Adaptation [CVPR 2024]
• Characterizing and Overcoming the Greedy Nature of Learning in Multi-modal Deep Neural Networks [ICML 2022]
• Adaptive Mask Co-Optimization for Modal Dependence in Multimodal Learning [ICASSP 2024]

Objective

---

• MMCosine: Multi-Modal Cosine Loss Towards Balanced Audio-Visual Fine-Grained Learning [ICASSP 2023]
• On Uni-Modal Feature Learning in Supervised Multi-Modal Learning [ICML 2023]
• Multimodal Pre-Training with Self-Distillation for Product Understanding in E-Commerce [WSDM 2023]
• Calibrating Multimodal Learning [ICML 2023]
• MMPareto: Boosting Multimodal Learning with Innocent Unimodal Assistance [ICML 2024]
• What Makes Training Multi-Modal Classification Networks Hard? [CVPR 2020]
• Facilitating Multimodal Classification via Dynamically Learning Modality Gap [NIPS 2024]

Optimization

---

• Balanced Multimodal Learning via On-the-fly Gradient Modulation [CVPR2022]
• Boosting Multi-modal Model Performance with Adaptive Gradient Modulation [ICCV 2023]
• PMR: Prototypical Modal Rebalance for Multimodal Learning [CVPR 2023]
• Diagnosing and Re-learning for Balanced Multimodal Learning [ECCV2024]
• ReconBoost: Boosting Can Achieve Modality Reconcilement. [ICML 2024]

BML in MLLMs

• MokA: Multimodal Low-Rank Adaptation for MLLMs [NeurIPS 2025]
• Fork-Merge Decoding: Enhancing Multimodal Understanding in Audio-Visual Large Language Models [arXiv 2025]
• Generalizing from SIMPLE to HARD Visual Reasoning: Can We Mitigate Modality Imbalance in VLMs? [arXiv 2025]