A Survey of Multimodal Federated Learning: Background, Applications, and Perspectives

This repository provides a comprehensive collection of papers focused on Multimodal Federated Learning (MMFL), with the primary researches already discussed in our latest review. This repository will continue to be updated, and we welcome you to give it a STAR⭐️.

Multimodal Federated Learning is a collaborative training process involving multiple clients, each with diverse modality settings and data, conducting learning tasks without disclosing their local raw data.

Table of Contents

• Survey
• Unifying Achitectures
• Applications
• Multimodal Datasets

Survey

Unifying Achitectures

Applications

🏃‍♂️Human Activity Recognition

A unified framework for multi-modal federated learning

B Xiong, X Yang, F Qi, C Xu

Neurocomputing, 202204 PUB

Multimodal federated learning on iot data

Y Zhao, P Barnaghi, H Haddadi

IoTDI, 202205 PUB

Cross-modal federated human activity recognition via modality-agnostic and modality-specific representation learning

X Yang, B Xiong, Y Huang, C Xu

AAAI, 2022 PUB

Towards optimal multi-modal federated learning on non-iid data with hierarchical gradient blending

S Chen, B Li

INFOCOM, 2022 PUB

FL-FD: Federated learning-based fall detection with multimodal data fusion

P Qi, D Chiaro, F Piccialli

Information Fusion, 202311 PUB

Cross-Modal Federated Human Activity Recognition

X Yang, B Xiong, Y Huang, C Xu

IEEE Transactions on Pattern Analysis and Machine Intelligence, 202402 PUB

FedMEKT: Distillation-based Embedding Knowledge Transfer for Multimodal Federated Learning

HQ Le, MNH Nguyen, CM Thwal, Y Qiao, C Zhang, CS Hong

arXiv, 2023 arXiv

👩🏿‍⚕Medical Diagnosis

Multimodal melanoma detection with federated learning

BLY Agbley, J Li, AU Haq, EK Bankas, S Ahmad, IO Agyemang, D Kulevome, WD Ndiaye, B Cobbinah, S Latipova

ICCWAMTIP, 202112 PUB

Harmony: Heterogeneous multi-modal federated learning through disentangled model training

X Ouyang, Z Xie, H Fu, S Cheng, L Pan, N Ling, G Xing, J Zhou, J Huang

MobiSys, 2023 PUB

A federated learning system with data fusion for healthcare using multi-party computation and additive secret sharing

T Muazu, Y Mao, AU Muhammad, M Ibrahim, UMM Kumshe, O Samuel

Computer Communications, 202402 PUB

Medical report generation based on multimodal federated learning

J Chen, R Pan

Computerized Medical Imaging and Graphics, 202404 PUB

Federated Modality-Specific Encoders and Multimodal Anchors for Personalized Brain Tumor Segmentation

Q Dai, D Wei, H Liu, J Sun, L Wang, Y Zheng

AAAI, 2024 PUB

🔍️Cross-modal Retrieval

FedCMR: Federated cross-modal retrieval

L Zong, Q Xie, J Zhou, P Wu, X Zhang, B Xu

SIGIR, 2021 PUB

Multimodal Federated Learning via Contrastive Representation Ensemble

Q Yu, Y Liu, Y Wang, K Xu, J Liu

ICLR, 2023 PUB

💬Visual Question Answer

Think locally, act globally: Federated learning with local and global representations

PP Liang, T Liu, L Ziyin, NB Allen, RP Auerbach, D Brent, R Salakhutdinov, LP Morency

NeurIPS, 2019 arXiv

Federated learning for vision-and-language grounding problems

F Liu, X Wu, S Ge, W Fan, Y Zou

AAAI, 2020 PUB

Multimodal Federated Learning with Missing Modality via Prototype Mask and Contrast

G Bao, Q Zhang, D Miao, Z Gong, L Hu

arXiv, 2024 arXiv

🤣Emotion Recognition

Federated Meta-Learning for Emotion and Sentiment Aware Multi-modal Complaint Identification

A Singh, S Chandrasekar, S Saha, T Sen

EMNLP, 2023 PUB

Enhancing Emotion Recognition through Federated Learning: A Multimodal Approach with Convolutional Neural Networks

N Simić, S Suzić, N Milošević, V Stanojev, T Nosek, B Popović, D Bajović

Applied Sciences, 202402 PUB

FedCMD: A Federated Cross-Modal Knowledge Distillation for Drivers Emotion Recognition

S Bano, N Tonellotto, P Cassarà, A Gotta

ACM Transactions on Intelligent Systems and Technology, 202405 PUB

🧙‍♂️Prompt Learning and Model Finetuning

Pfedprompt: Learning personalized prompt for vision-language models in federated learning

T Guo, S Guo, J Wang

WWW, 2023 PUB

Global and Local Prompts Cooperation via Optimal Transport for Federated Learning

H Li, W Huang, J Wang, Y Shi

CVPR, 2024 arXiv

Feddat: An approach for foundation model finetuning in multi-modal heterogeneous federated learning

H Chen, Y Zhang, D Krompass, J Gu, V Tresp

AAAI, 2024 PUB

📡Internet of Things

Fedfusion: Manifold driven federated learning for multi-satellite and multi-modality fusion

DX Li, W Xie, Y Li, L Fang

Geoscience and Remote Sensing, 2023 PUB

Autofed: Heterogeneity-aware federated multimodal learning for robust autonomous driving

T Zheng, A Li, Z Chen, H Wang, J Luo

ACM MobiCom, 2023 PUB

FedUSL: A Federated Annotation Method for Driving Fatigue Detection based on Multimodal Sensing Data

S Yu, Q Yang, J Wang, C Wu

ACM Transactions on Sensor Networks, 202403 PUB

🏷️Image Classification

Fedclip: Fast generalization and personalization for clip in federated learning

W Lu, X Hu, J Wang, X Xie

Data Engineering Bulletin, 2023 arXiv

Multimodal Datasets

Citation

If you find the listing and survey useful for your work, please cite the paper:

@article{pan2024survey,
  title={A survey of multimodal federated learning: background, applications, and perspectives},
  author={Pan, Hao and Zhao, Xiaoli and He, Lipeng and Shi, Yicong and Lin, Xiaogang},
  journal={Multimedia Systems},
  volume={30},
  number={4},
  pages={222},
  year={2024},
  publisher={Springer}
}