threegold116/Awesome-Omni-MLLMs

This is for ACL 2025 Findings Paper: From Specific-MLLMs to Omni-MLLMs: A Survey on MLLMs Aligned with Multi-modalitiesModels

104

81 commits

updated Mar 22, 2026

See the code

README

From Specific-MLLMs to Omni-MLLMs: A Survey on MLLMs Aligned with Multi-modalities

Shixin Jiang^1∗, Jiafeng Liang¹, Jiyuan Wang¹, Xuan Dong¹, Heng Chang³, Weijiang Yu³, Jinhua Du⁴, Ming Liu^1†,2, Bing Qin^1,2

¹Harbin Institute of Technology, Harbin, China

²Peng Cheng Laboratory, Shenzhen, China

³Huawei Inc., Shenzhen, China

⁴Huawei London Research Centre, London, UK

This repository contains the resources for arxiv paper From Specific-MLLMs to Omni-MLLMs: A Survey on MLLMs Aligned with Multi-modalities

taxonomy

For more details, please refer to the paper: From Specific-MLLMs to Omni-MLLMs: A Survey on MLLMs Aligned with Multi-modalities.

📢 Updates

2025/05/16 Our paper is accepted as ACL2025 Findings 🎉.
2025/02/28 We have updated 23 papers in the reading list 📚.
2025/02/20 We have updated 55 papers first in the reading list 📚.
2025/02/15 The second version of our paper has been released, check it on arxiv 🎓.
2024/12/16 The first version of our paper is available on arxiv 🎓.

This reading list will be updated periodically, and if you have any suggestions or find some we missed, feel free to contact us! You can submit an issue or send an email (sxjiang@ir.hit.edu.cn).

🗂️ Tables of Contents

🧠Awesome Omni-MLLMs
🧪Cross-Modality Benchmark
- 📊Comprehensive Understanding Benchmark
- 📉Down-Streaming Understanding Benchmark
🔍Uni-Modality Benchmark
- 🔄Any2Any Benchmark

🧠Awesome Omni-MLLMs

Omni-MLLM: The MLLMs which can handle more than 2 extra-linguistic modalities.

🌿Multi-branch Continuous Omni-MLLMs

Title	Venue	Date	Code	Modalities	Cross(Omni)-modal Understanding	Cross(Omni)-modal Generation
OmniGAIA: Towards Native Omni-Modal AI Agents	Arxiv	2026-02-28	Github	Visual/Audio/Text ➜Text	✅	❌
ThinkOmni: Lifting Textual Reasoning to Omni-modal Scenarios via Guidance Decoding	Arxiv	2026-02-28	--	Visual/Audio/Text ➜Text	✅	❌
OmniVideo-R1: Reinforcing Audio-visual Reasoning with Query Intention and Modality Attention	Arxiv	2026-02-15	--	Visual/Audio/Text ➜Text	✅	❌
Omni-Safety under Cross-Modality Conflict: Vulnerabilities, Dynamics Mechanisms and Efficient Alignment	Arxiv	2026-02-10	Github	Visual/Audio/Text ➜Text	✅	❌
ChronusOmni: Improving Time Awareness of Omni Large Language Models	Arxiv	2025-12-10	Github	Visual/Audio/Text ➜Text	✅	❌
Omni-AutoThink: Adaptive Multimodal Reasoning via Reinforcement Learning	Arxiv	2025-12-03	Github	Visual/Audio/Text ➜Text	✅	❌
Uni-MoE-2.0-Omni: Scaling Language-Centric Omnimodal Large Model with Advanced MoE, Training and Data	Arxiv	2025-11-16	Github	Visual/Audio/Text ➜Text/Audio/Visual	✅	✅
Omni-AVSR: Towards Unified Multimodal Speech Recognition with Large Language Models	Arxiv	2025-11-10	Github	Visual/Audio/Text ➜Text	✅	❌
LongCat-Flash-Omni Technical Report	Arxiv	2025-10-31	Github	Visual/Audio/Text ➜Text/Audio	✅	✅
Ming-Flash-Omni: A Sparse, Unified Architecture for Multimodal Perception and Generation	Arxiv	2025-10-28	Github	Visual/Audio/Text ➜Text/Visual/Audio	✅	✅
Omni-Reward: Towards Generalist Omni-Modal Reward Modeling with Free-Form Preferences	Arxiv	2025-10-27	Github	Visual/Audio/Text ➜Scalar	✅	❌
RoboOmni: Proactive Robot Manipulation in Omni-modal Context	Arxiv	2025-10-27	Github	Visual/Audio/Text ➜Text/Audio/Action	✅	✅
OmniVinci: Enhancing Architecture and Data for Omni-Modal Understanding LLM	Arxiv	2025-10-17	Github	Visual/Audio/Text ➜Text/Audio	✅	✅
InteractiveOmni: A Unified Omni-modal Model for Audio-Visual Multi-turn Dialogue	Arxiv	2025-10-15	Github	Visual/Audio/Text ➜Text/Audio	✅	✅
MoME: Mixture of Matryoshka Experts for Audio-Visual Speech Recognition	NIPS	2025-10-05	--	Visual/Audio/Text ➜Text	✅	❌
Qwen3-Omni Technical Report	arXiv	2025-09-26	Github	Visual/Audio/Text ➜Text/Audio	✅	✅
ShizhenGPT: Towards Multimodal LLMs for Traditional Chinese Medicine	Arxiv	2025-08-20	Github	Visual/Audio/Text ➜Text	✅	❌
Think Before You Segment: An Object-aware Reasoning Agent for Referring Audio-Visual Segmentation	Arxiv	2025-08-06	--	Visual/Audio/Text ➜Text	✅	❌
VeOmni: Scaling Any Modality Model Training with Model-Centric Distributed Recipe Zoo	Arxiv	2025-08-04	--	Visual/Audio/Text ➜Text	✅	✅
AURORA: Augmented Understanding via Structured Reasoning and Reinforcement Learning for Reference Audio-Visual Segmentation	Arxiv	2025-08-04	--	Visual/Audio/Text ➜Text	✅	❌
Towards Omnimodal Expressions and Reasoning in Referring Audio-Visual Segmentation	ICCV	2025-07-30	--	Visual/Audio/Text ➜Text	✅	❌
ARC-Hunyuan-Video-7B: Structured Video Comprehension of Real-World Shorts	Arxiv	2025-07-29	--	Visual/Audio/Text ➜Text	✅	❌
SpeakerVid-5M: A Large-Scale High-Quality Dataset for Audio-Visual Dyadic Interactive Human Generation	Arxiv	2025-07-15	--	Visual/Audio/Text ➜Text	✅	✅
HumanOmniV2: From Understanding to Omni-Modal Reasoning with Context	Arxiv	2025-06-26	Github	Visual/Audio/Text ➜Text	✅	❌
video-SALMONN 2: Captioning-Enhanced Audio-Visual Large Language Models	Arxiv	2025-06-05	--	Visual/Audio/Text ➜Text	✅	❌
AV-Reasoner: Improving and Benchmarking Clue-Grounded Audio-Visual Counting for MLLMs	Arxiv	2025-06-05	Github	Visual/Audio/Text ➜Text	✅	❌
Is Extending Modality The Right Path Towards Omni-Modality?	Arxiv	2025-06-02	Github	Visual/Audio/Text ➜Text	✅	❌
Unifying Multimodal Large Language Model Capabilities and Modalities via Model Merging	Arxiv	2025-05-26	Github	Visual/Audio/Text ➜Text	✅	❌
Omni-R1: Reinforcement Learning for Omnimodal Reasoning via Two-System Collaboration	Arxiv	2025-05-26	Github	Visual/Audio/Text ➜Text	✅	❌
Unifying Multimodal Large Language Model Capabilities and Modalities via Model Merging	Arxiv	2025-05-26	Github	Visual/Audio/Text ➜Text	✅	❌
Watch and Listen: Understanding Audio-Visual-Speech Moments with Multimodal LLM	Arxiv	2025-05-23	--	Visual/Audio/Text ➜Text	✅	❌
Ming-Omni: A Unified Multimodal Model for Perception and Generation	Arxiv	2025-05-07	Github	Visual/Audio/Text ➜Text/Audio/Visual	✅	✅
EchoInk-R1: Exploring Audio-Visual Reasoning in Multimodal LLMs via Reinforcement Learning	Arxiv	2025-05-07	Github	Visual/Audio/Text ➜Text	✅	❌
Multimodal Long Video Modeling Based on Temporal Dynamic Context	Arxiv	2025-04-14	Github	Visual/Audio/Text ➜Text	✅	❌
Capybara-OMNI: An Efficient Paradigm for Building Omni-Modal Language Models	ICLR Workshop	2025-04-10	Github	Visual/Audio/Text ➜Text	✅	❌
FortisAVQA and MAVEN: a Benchmark Dataset and Debiasing Framework for Robust Multimodal Reasoning	Arxiv	2025-04-01	Github	Visual/Audio/Text ➜Text	✅	❌
OrchMLLM: Orchestrate Multimodal Data with Batch Post-Balancing to Accelerate Multimodal Large Language Model Training	arXiv	2025-03-31	-	Visual/Audio/Text ➜Text	✅	❌
OmniMMI: A Comprehensive Multi-modal Interaction Benchmark in Streaming Video Contexts	CVPR	2025-03-28	Github	Visual/Audio/Text ➜Text	✅	❌
DeepSound-V1: Start to Think Step-by-Step in the Audio Generation from Videos	arXiv	2025-03-27	-	Visual/Audio/Text ➜Text/Audio	✅	✅
Qwen2.5-Omni Technical Report	arXiv	2025-03-27	Github	Visual/Audio/Text ➜Text/Audio	✅	✅
Vision-Speech Models: Teaching Speech Models to Converse about Images	arXiv	2025-03-20	Github	Visual/Audio/Text ➜Text/Audio	✅	✅
ViSpeak: Visual Instruction Feedback in Streaming Videos	arXiv	2025-03-17	Github	Visual/Audio/Text ➜Text/Audio	✅	✅
Crab: A Unified Audio-Visual Scene Understanding Model with Explicit Cooperation	CVPR	2025-03-17	Github	Visual/Audio/Text ➜Text	✅	❌
MMS-LLaMA: Efficient LLM-based Audio-Visual Speech Recognition with Minimal Multimodal Speech Tokens	arXiv	2025-03-14	Github	Visual/Audio/Text ➜Text	✅	❌
PhysVLM: Enabling Visual Language Models to Understand Robotic Physical Reachability	arXiv	2025-03-11	-	Visual/3D/Text ➜Text	✅	❌
Adaptive Audio-Visual Speech Recognition via Matryoshka-Based Multimodal LLMs	arXiv	2025-03-09	-	Visual/Audio/Text ➜Text	✅	❌
R1-Omni: Explainable Omni-Multimodal Emotion Recognition with Reinforcing Learning	arXiv	2025-03-07	Github	Visual/Audio/Text ➜Text	✅	❌
EgoLife: Towards Egocentric Life Assistant	CVPR	2025-03-05	Github	Visual/Audio/Text ➜Text	✅	❌
Phi-4-Mini Technical Report: Compact yet Powerful Multimodal Language Models via Mixture-of-LoRAs	arXiv	2025-03-03	Github	Visual/Audio/Text ➜Text	✅	❌
Nexus-O: An Omni-Perceptive And -Interactive Model for Language, Audio, And Vision	arXiv	2025-02-26		Visual/Audio/Text ➜Text/Audio	✅	✅
Megrez-Omni Technical Report	arXiv	2025-02-19	Github	Visual/Audio/Text ➜Text	✅	❌
video-SALMONN-o1: Reasoning-enhanced Audio-visual Large Language Model	ICML	2025-02-17	Github	Visual/Audio/Text ➜Text	✅	❌
Towards Multimodal Empathetic Response Generation: A Rich Text-Speech-Vision Avatar-based Benchmark	WWW	2025-02-07	-	Visual/Audio/Text ➜Text/Visual/Audio	✅	✅
M2-omni: Advancing Omni-MLLM for Comprehensive Modality Support with Competitive Performance	arXiv	2025-02-06	-	Visual/Audio/Text ➜Text/Visual/Audio	✅	✅
Ola: Pushing the Frontiers of Omni-Modal Language Model with Progressive Modality Alignment	arXiv	2025-02-06	Github	Visual/Audio/Text ➜Text/Audio	✅	✅
AGAV-Rater: Adapting Large Multimodal Model for AI-Generated Audio-Visual Quality Assessment	arXiv	2025-01-30	Github	Visual/Audio/Text ➜Text	✅	❌
HumanOmni: A Large Vision-Speech Language Model for Human-Centric Video Understanding	arXiv	2025-01-25	Github	Visual/Audio/Text ➜Text	✅	❌
MiniCPM-o 2.6: A GPT-4o Level MLLM for Vision, Speech, and Multimodal Live Streaming on Your Phone	blog	2025-01-24	Github	Visual/Audio/Text ➜Text/Audio	✅	✅
Omni-Emotion: Extending Video MLLM with Detailed Face and Audio Modeling for Multimodal Emotion Analysis	arXiv	2025-01-16	Github	Visual/Audio/Text ➜Text	✅	❌
OpenOmni: A Fully Open-Source Omni Large Language Model with Real-time Self-Aware Emotional Speech Synthesis	arXiv	2025-01-09	Github	Visual/Audio/Text ➜Text/Audio	✅	✅
Empathetic Response in Audio-Visual Conversations Using Emotion Preference Optimization and MambaCompressor	arXiv	2024-12-23	-	Visual/Audio/Text ➜Text	✅	❌
VITA-1.5: Towards GPT-4o Level Real-Time Vision and Speech Interaction	arXiv	2024-12-20	Github	Visual/Audio/Text ➜Text/Audio	✅	✅
Lyra: An Efficient and Speech-Centric Framework for Omni-Cognition	arXiv	2024-12-12	Github	Visual/Audio/Text ➜Text	✅	❌
MuMu-LLaMA: Multi-modal Music Understanding and Generation via Large Language Models	arXiv	2024-12-09	Github	Visual/Audio/Text ➜Text/Audio	✅	✅
LongVALE: Vision-Audio-Language-Event Benchmark Towards Time-Aware Omni-Modal Perception of Long Videos	CVPR	2024-11-29	-	Visual/Audio/Text ➜Text	✅	❌
SAVEn-Vid: Synergistic Audio-Visual Integration for Enhanced Understanding in Long Video Context	arXiv	2024-11-25	Github	Visual/Audio/Text ➜Text	✅	❌
Spider: Any-to-Many Multimodal LLM	arXiv	2024-11-14	-	Visual/Audio/Text ➜Text/Visual/Audio	✅	✅
Towards Multi-Modal Mastery: A 4.5B Parameter Truly Multi-Modal Small Language Model	arXiv	2024-11-08	-	Visual/Audio/Text ➜Text/Audio	✅	✅
CAD-MLLM: Unifying Multimodality-ConditionedCAD Generation With MLLM	arXiv	2024-11-07	Github	Visual/3D/Text ➜Text/3D	✅	✅
LLMs Can Evolve Continually on Modality for X-Modal Reasoning	NeurIPS	2024-10-26	Github	Visual/Audio/3D/Depth/Text ➜Text	✅	❌
OMCAT Omni Context Aware Transformer	arXiv	2024-10-15	-	Visual/Audio/Text ➜Text	✅	❌
Mini-Omni2: Towards Open-source GPT-4o with Vision, Speech and Duplex Capabilities	NeurIPS	2024-10-15	Github	Visual/Audio/Text ➜Text/Audio	✅	✅
Aligned Better, Listen Better For Audio-Visual Large Language Models	ICLR	2024-10-13	-	Visual/Audio/Text ➜Text	✅	❌
Baichuan-Omni Technical Report	arXiv	2024-10-11	Github	Visual/Audio/Text ➜Text	✅	❌
Efficient Generative Multimodal Integration (EGMI): Enabling Audio Generation from Text-Image Pairs through Alignment with Large Language Models	NeurIPS Workshop	2024-10-11	-	Visual/Audio/Text ➜Text/Audio	✅	✅
Enhancing Multimodal LLM for Detailed and Accurate Video Captioning using Multi-Round Preference Optimization	arXiv	2024-10-09	-	Visual/Audio/Text ➜Text	✅	❌
Large Language Models Are Strong Audio-Visual Speech Recognition Learners	ICASSP	2024-09-18	-	Visual/Audio/Text ➜Text	✅	❌
VITA: Towards Open-Source Interactive Omni Multimodal LLM	arXiv	2024-08-05	Github	Visual/Audio/Text ➜Text/Audio	✅	✅
UnifiedMLLM: Enabling Unified Representation for Multi-modal Multi-tasks With Large Language Model	arXiv	2024-08-03	Github	Visual/Audio/Text ➜Text/Visual/Audio	✅	✅
InternOmni: Extending InternVL with Audio Modality	blog	2024-07-31	-	Visual/Audio/Text ➜Text	✅	❌
Meerkat: Audio-Visual Large Language Model for Grounding in Space and Time	arXiv	2024-07-01	Github	Visual/Audio/Text ➜Text	✅	❌
video-SALMONN: Speech-Enhanced Audio-Visual Large Language Models	arXiv	2024-06-22	Github	Visual/Audio/Text ➜Text	✅	❌
Emotion-LLaMA: Multimodal Emotion Recognition and Reasoning with Instruction Tuning	NIPS	2024-06-17	Github	Visual/Audio/Text ➜Text	✅	❌
VideoLLaMA 2 Advancing Spatial-Temporal Modeling and Audio Understanding in Video-LLMs	arXiv	2024-06-11	Github	Visual/Audio/Text ➜Text	✅	❌
Recognizing Everything from All Modalities at Once: Grounded Multimodal Universal Information Extraction	ACL	2024-06-06	Github	Visual/Audio/Text ➜Text	✅	❌
X-VILA: Cross-Modality Alignment for Large Language Model	arXiv	2024-05-29	-	Visual/Audio/Text ➜Text/Visual/Audio	✅	✅
Uni-MoE: Scaling Unified Multimodal LLMs with Mixture of Experts	TPAMI	2024-05-18	Github	Visual/Audio/Text ➜Text	✅	❌
Querying as Prompt: Parameter-Efficient Learning for Multimodal Language Model	CVPR	2024-05-06	Github	Visual/Audio/Text ➜Text	✅	❌
WorldGPT: Empowering LLM as Multimodal World Model	AAAI	2024-04-28	Github	Visual/Audio/Text ➜Text/Visual/Audio	✅	✅
Empowering LLMs with Pseudo-Untrimmed Videos for Audio-Visual Temporal Understanding	arXiv	2024-03-24	-	Visual/Audio/Text ➜Text	✅	❌
CAT : Enhancing Multimodal Large Language Model to Answer Questions in Dynamic Audio-Visual Scenarios	ECCV	2024-03-04	Github	Visual/Audio/Text ➜Text	✅	❌
Model Composition for Multimodal Large Language Models	ACL	2024-02-18	Github	Visual/Audio/3D/Text ➜Text	✅	❌
GroundingGPT: Language Enhanced Multi-modal Grounding Model	ACL	2024-02-11	Github	Visual/Audio/Text ➜Text	✅	❌
MLLM-Tool: A Multimodal Large Language Model For Tool Agent Learning	ECCV	2024-01-19	Github	Visual/Audio/Text ➜Text/Visual	✅	✅
MultiPLY: A Multisensory Object-Centric Embodied Large Language Model in 3D World	CVPR	2024-01-16	Github	Visual/3D/Touch/Text ➜Text	✅	❌
ModaVerse: Efficiently Transforming Modalities with LLMs	CVPR	2024-01-12	Github	Visual/Audio/Text ➜Text/Visual/Audio	✅	✅
Unified-IO 2: Scaling Autoregressive Multimodal Models with Vision, Language, Audio, and Actions	CVPR	2023-12-28	Github	Visual/Audio/Text ➜Text/Visual/Audio	✅	✅
M3DBench: Towards Omni 3D Assistant with Interleaved Multi-modal Instructions	ECCV	2023-12-17	Github	Visual/3D/Text ➜Text	✅	❌
DriveMLM: Aligning Multi-Modal Large Language Models with Behavioral Planning States for Autonomous Driving	arXiv	2023-12-14	Github	Visual/3D/Text ➜Text	✅	❌
Audio-Visual LLM for Video Understanding	arXiv	2023-12-11	-	Visual/Audio/Text ➜Text	✅	❌
X-InstructBLIP: A Framework for aligning X-Modal instruction-aware representations to LLMs and Emergent Cross-modal Reasoning	ECCV	2023-11-30	Github	Visual/Audio/3D/Text ➜Text	✅	❌
Octavius: Mitigating Task Interference in MLLMs via LoRA-MoE	ICLR	2023-11-05	Github	Visual/3D/Text ➜Text	✅	❌
Fine-grained Audio-Visual Joint Representations for Multimodal Large Language Models	arXiv	2023-10-09	Github	Visual/Audio/Text ➜Text	✅	❌
AnyMAL: An Efficient and Scalable Any-Modality Augmented Language Model	EMNLP industry	2023-09-27	Github	Visual/Audio/IMU/Text ➜Text	✅	❌
NExT-GPT: Any-to-Any Multimodal LLM	ICML	2023-09-11	Github	Visual/Audio/Text ➜Text/Visual/Audio	✅	✅
BuboGPT: Enabling Visual Grounding in Multi-Modal LLMs	arXiv	2023-07-17	Github	Visual/Audio/Text ➜Text	✅	❌
Macaw-LLM: Multi-Modal Language Modeling with Image, Audio, Video, and Text Integration	arXiv	2023-06-15	Github	Visual/Audio/Text ➜Text	✅	❌
LAMM: Language-Assisted Multi-Modal Instruction-Tuning Dataset, Framework, and Benchmark	NerIPS	2023-06-13	Github	Visual/3D/Text ➜Text	✅	❌
Video-LLaMA: An Instruction-tuned Audio-Visual Language Model for Video Understanding	EMNLP demo	2023-06-05	Github	Visual/Audio/Text ➜Text	✅	❌
ChatBridge: Bridging Modalities with Large Language Model as a Language Catalyst	arXiv	2023-05-25	Github	Visual/Audio/Text ➜Text	✅	❌
X-LLM: Bootstrapping Advanced Large Language Models by Treating Multi-Modalities as Foreign Languages	arXiv	2023-05-07	Github	Visual/Audio/Text ➜Text	✅	❌
eP-ALM: Efficient Perceptual Augmentation of Language Models	ICCV	2023-03-20	Github	Visual/Audio/Text ➜Text	✅	❌

🌲Uni-branch Continuous Omni-MLLMs

Title	Venue	Date	Code	Modalities	Cross(Omni)-modal Understanding	Cross(Omni)-modal Generation
LLMs Can Evolve Continually on Modality for X-Modal Reasoning	NeurIPS	2024-10-26	Github	Visual/Audio/3D/Depth/Text ➜Text/Audio	✅	❌
Efficient Generative Multimodal Integration (EGMI): Enabling Audio Generation from Text-Image Pairs through Alignment with Large Language Models	NeurIPS Workshop	2024-10-11	-	Visual/Audio/Text ➜Text/Visual/Audio	✅	✅
EmpathyEar: An Open-source Avatar Multimodal Empathetic Chatbot	ACL Demo	2024-06-21	Github	Visual/Audio/Text ➜Text/Visual/Audio	✅	✅
WorldGPT: Empowering LLM as Multimodal World Model	MM	2024-04-28	Github	Visual/Audio/Text ➜Text/Visual/Audio	✅	✅
A Touch, Vision, and Language Dataset for Multimodal Alignment	ICML	2024-02-20	Github	Visual/Touch/Text ➜Text	✅	❌
CREMA: Generalizable and Efficient Video-Language Reasoning via Multimodal Modular Fusion	ICLR	2024-02-08	Github	Visual/Audio/3D/Depth/ Thremal/Text➜Text	✅	❌
OneLLM: One Framework to Align All Modalities with Language	CVPR	2023-11-30	Github	Visual/Audio/3D/Depth/ IMU/Text➜Text	✅	❌
CoDi-2: In-Context, Interleaved, and Interactive Any-to-Any Generation	CVPR	2023-11-30	Github	Visual/Audio/Text ➜Text/Visual/Audio	✅	✅
ImageBind-LLM: Multi-modality Instruction Tuning	arXiv	2023-09-11	Github	Visual/Audio/3D/ Text➜Text	✅	❌
PandaGPT: One Model To Instruction-Follow Them All	tllm	2023-05-25	Github	Visual/Audio/3D/Depth/ Thremal/Text➜Text	✅	❌

🧩Diserect Omni-MLLMs

Title	Venue	Date	Code	Modalities	Cross(Omni)-modal Understanding	Cross(Omni)-modal Generation
NExT-OMNI: Towards Any-to-Any Omnimodal Foundation Models with Discrete Flow Matching	Arxiv	2025-10-15	-	Visual/Audio/Text ➜Text/Visual/Audio	✅	✅
SOLAMI: Social Vision-Language-Action Modeling for Immersive Interaction with 3D Autonomous Characters	Arxiv	2024-12-29	-	Visual/Audio/Motion/ Text➜Text/Audio/Motion	✅	✅
LLM Gesticulator: Leveraging Large Language Models for Scalable and Controllable Co-Speech Gesture Synthesis	Arxiv	2024-10-06	-	Visual/Audio/Motion/ Text➜Text/Audio/Motion	✅	✅
MIO: A Foundation Model on Multimodal Tokens	Arxiv	2024-09-26	Github	Visual/Audio/Text ➜Text/Visual/Audio	✅	✅
OccLLaMA: An Occupancy-Language-Action Generative World Model for Autonomous Driving	Arxiv	2024-09-05	-	Action/3D/Text ➜Text/3D/Action	✅	✅
M3GPT: An Advanced Multimodal, Multitask Framework for Motion Comprehension and Generation	NeurIPS	2024-02-19	Github	Audio/Motion/Text ➜Text/Audio/Motion	✅	✅
AnyGPT: Unified Multimodal LLM with Discrete Sequence Modeling	ACL	2024-02-19	Github	Visual/Audio/Text ➜Text/Visual/Audio	✅	✅
TEAL: Tokenize and Embed ALL for Multi-modal Large Language Models	Arxiv	2023-11-08	-	Visual/Audio/Text ➜Text	✅	❌

🔀Hybrid Omni-MLLMs

Title	Venue	Date	Code	Demo	Cross(Omni)-modal Understanding	Cross(Omni)-modal Generation
Stream-Omni: Simultaneous Multimodal Interactions with Large Language-Vision-Speech Model	arXiv	2025-06-02	Github	Visual/Audio/Text ➜Text/Audio	✅	✅
RoboEgo System Card: An Omnimodal Model with Native Full Duplexity	arXiv	2025-06-02	Github	Visual/Audio/Text ➜Text/Audio	✅	✅
ShapeLLM-Omni: A Native Multimodal LLM for 3D Generation and Understanding	arXiv	2025-06-02	--	Visual/3D/Text ➜Text/3D	✅	✅
Baichuan-Omni-1.5 Technical Report	arXiv	2025-01-26	Github	Visual/Audio/Text ➜Text/Audio	✅	✅
From Multimodal LLMs to Generalist Embodied Agents: Methods and Lessons	Arxiv	2024-12-11	-	Visual/Action/Text ➜Text/Action	✅	✅
EMOVA: Empowering Language Models to See, Hear and Speak with Vivid Emotions	CVPR	2024-09-26	Github	Visual/Audio/Text ➜Text/Audio	✅	✅
Grounding Multimodal Large Language Models in Actions	NeurIPS	2024-06-12	-	Visual/Action/Text ➜Text/Action	✅	✅
An Embodied Generalist Agent in 3D World	ICML	2023-11-18	Github	Visual/3D/Action/ Text➜Text	✅	❌

🧪Cross-Modality Benchmark

📌Input-Output format: X+Y+Text➜Text or X+Y+Text➜X

📊Comprehensive Understanding Benchmark

Title	Name	Date	Download	Modality
FutureOmni: Evaluating Future Forecasting from Omni-Modal Context for Multimodal LLMs	FutureOmni	2026-01	--	Video,Audio, Text
LiViBench: An Omnimodal Benchmark for Interactive Livestream Video Understanding	LiViBench	2026-01	--	Video,Audio, Text
OmniVideoBench: Towards Audio-Visual Understanding Evaluation for Omni MLLMs	OmniVideoBench	2025-10-12	Huggingface	Video,Audio, Text
Unobench: A unified benchmark for exploring the compositional law between uni-modal and omni-modal in omni models	UNO-Bench	2025-10	Github	Video,Audio, Text
VoiceAssistant-Eval: Benchmarking AI Assistants across Listening, Speaking, and Viewing	VoiceAssistant-Eval	2025-09-26	Huggingface	Image,Audio, Text
HumanSense: From Multimodal Perception to Empathetic Context-Aware Responses through Reasoning MLLMs	HumanSense	2025-08-14	Github	Video,Audio, Text
VGGSounder: Audio-Visual Evaluations for Foundation Models	VGGSounder	2025-08-14	Huggingface	Video,Audio, Text
OmniEval: A Benchmark for Evaluating Omni-modal Models with Visual, Auditory, and Textual Inputs	OmniEval	2025-06-26	Huggingface	Video,Audio, Text
Daily-Omni: Towards Audio-Visual Reasoning with Temporal Alignment across Modalities	Daily-Omni	2025-05-23	--	Video,Audio, Text
AURELIA: Test-time Reasoning Distillation in Audio-Visual LLMs	AVReasonBench	2025-03-29	--	Video,Audio, Text
OmniMMI: A Comprehensive Multi-modal Interaction Benchmark in Streaming Video Contexts	OmniMMI	2025-03-28	Huggingface	Video,Audio, Text
MAVERIX: Multimodal Audio-Visual Evaluation Reasoning IndeX	MAVERIX	2025-03-27	--	Video,Audio, Text
ACVUBench: Audio-Centric Video Understanding Benchmark	ACVUBench	2025-03-25	--	Video,Audio, Text
DAVE : Diagnostic benchmark for Audio Visual Evaluation	DAVE	2025-03-12	--	Video,Audio, Text
video-SALMONN-o1: Reasoning-enhanced Audio-visual Large Language Model	AVRBench	2025-02-17	Huggingface	Video,Audio, Text
WorldSense: Evaluating Real-world Omnimodal Understanding for Multimodal LLMs	WorldSense	2025-02-06	Huggingface	Image,Video, Audio,Text
AVTrustBench: Assessing and Enhancing Reliability and Robustness in Audio-Visual LLMs	AVTrustBench	2025-01-03	-	Image,Video, Audio,Text
Align Anything: Training All-Modality Models to Follow Instructions with Language Feedback	AMU	2024-12-20	-	Image,Video, Audio,Text
OmniBench: Towards The Future of Universal Omni-Language Models	OmniBench	2024-12-11	Huggingface	Image,Audio, Text
AV-Odyssey Bench: Can Your Multimodal LLMs Really Understand Audio-Visual Information?	AV-Odyssey Bench	2024-12-03	Huggingface	Image,Video, Audio,Text
AVHBench: A Cross-Modal Hallucination Benchmark for Audio-Visual Large Language Models	AVHBench	2024-10-23	Google Driver Google Driver	Image,Video, Audio,Text
LongVALE: Vision-Audio-Language-Event Benchmark Towards Time-Aware Omni-Modal Perception of Long Videos	LongVALE	2024-10-16	-	Image,Video, Audio,Text
The Curse of Multi-Modalities: Evaluating Hallucinations of Large Multimodal Models across Language, Visual, and Audio	Curse	2024-10-16	Huggingface	Image,Video, Audio,Text
OmnixR: Evaluating Omni-modality Language Models on Reasoning across Modalities	OmnixR	2024-10-16	Huggingface	Image,Video, Audio,Text
ShareGPT-4o: Comprehensive Multimodal Annotations With GPT-4o	ShareGPT-4o	2024-08-06	Huggingface	Image,Video, Audio,Text
video-SALMONN: Speech-Enhanced Audio-Visual Large Language Models	SAVE	2024-06-22	Huggingface	Image,Video, Audio,Text
Model Composition for Multimodal Large Language Models	MCUB	2024-02-20	-	Image,Audio, 3D,Text
A Touch, Vision, and Language Dataset for Multimodal Alignment	TVL Benchmark	2024-02-20	-	Image,Touch, Text
X-InstructBLIP: A Framework for Aligning Image, 3D, Audio, Video to LLMs and its Emergent Cross-Modal Reasoning	DisCRn	2023-11-30	Github	Image,Audio, 3D,Text

📉Down-Streaming Understanding Benchmark

Title	Name	Date	Download	Modality
ChronusOmni: Improving Time Awareness of Omni Large Language Models	ChronusAV	2025-12-10	Github	Video,Audio, Text
OmniGuard: Unified Omni-Modal Guardrails with Deliberate Reasoning	OmniGuard	2025-12-02	--	Video,Audio, Text
OmniPlay: Benchmarking Omni-Modal Models on Omni-Modal Game Playing	OmniPlay	2025-08-06	Github	Video,Audio, Text
Think Before You Segment: An Object-aware Reasoning Agent for Referring Audio-Visual Segmentation	R2-AVSBench	2025-08-06	Google Driver	Video,Audio, Image,Text
Towards Omnimodal Expressions and Reasoning in Referring Audio-Visual Segmentation	OmniAVS	2025-07-30	Huggingface	Video,Audio, Text
VideoMind: An Omni-Modal Video Dataset with Intent Grounding for Deep-Cognitive Video Understanding	VideoMind	2025-07-24	Huggingface	Video,Audio, Text
AnyCap Project: A Unified Framework, Dataset, and Benchmark for Controllable Omni-modal Captioning	AnyCap	2025-07-16	Huggingface	Video,Audio, Text
UGC-VideoCaptioner: An Omni UGC Video Detail Caption Model and New Benchmarks	UGC-VideoCap	2025-07-15	Huggingface	Video,Audio, Text
SAVVY: Spatial Awareness via Audio-Visual LLMs through Seeing and Hearing	SAVVY-Bench	2025-06-04	-	Video,Audio, Text
Multimodal Conversation Structure Understanding	TV-MMPC	2025-05-23	-	Video,Audio, Text
FortisAVQA and MAVEN: a Benchmark Dataset and Debiasing Framework for Robust Multimodal Reasoning	FortisAVQA	2025-04-01	-	Video,Audio, Text
Ref-AVS: Refer and Segment Objects in Audio-Visual Scenes	Ref-AVS	2024-07-15	Dataset	Image,Video, Audio,Text
Recognizing Everything from All Modalities at Once: Grounded Multimodal Universal Information Extraction	MUIE	2024-06-06	-	Image,Video, Audio,Text
Look, Listen, and Answer: Overcoming Biases for Audio-Visual Question Answering	MUSIC-AVQA-R	2024-04-18	-	Video,Audio, Text
Tackling Data Bias in MUSIC-AVQA: Crafting a Balanced Dataset for Unbiased Question-Answering	MUSIC-AVQA v2.0	2023-10-10	DropBox	Video,Audio, Text
VALOR: Vision-Audio-Language Omni-Perception Pretraining Model and Dataset	VALOR-32K	2023-04-17	BaiduDisk BaiduDisk	Video,Audio, Text
Dense-Localizing Audio-Visual Events in Untrimmed Videos: A Large-Scale Benchmark and Baseline	UnAV-100 test	2023-03-22	-	Video,Audio, Text
AVQA: A Dataset for Audio-Visual Question Answering on Videos	AVQA	2022-10-10	Dataset	Video,Audio, Text
Unified Multisensory Perception: Weakly-Supervised Audio-Visual Video Parsing	LLP	2022-07-21	--	Video,Audio, Text
Learning to Answer Questions in Dynamic Audio-Visual Scenarios	Music-AVQA	2022-05-26	Google Driver Github	Video,Audio, Text
Pano-AVQA: Grounded Audio-Visual Question Answering on 360 Videos	Pano-AVQA	2021-10-11	--	Video,Audio, Text
VGGSound: A Large-scale Audio-Visual Dataset	VGGSound	2020-04-29	Huggingface	Video,Audio, Text
Audio Visual Scene-Aware Dialog	AVSD	2019-01-25	Dataset	Video,Audio, Text

🔍Uni-Modality Benchmark

📌Input-Output format: X+Text➜Text or X+Text➜X

🔄Any2Any Benchmark

Title	Name	Date	Download	Modality
Judge Anything: MLLM as a Judge Across Any Modality	TaskAnything	2025-03-21	Huggingface	Video,Image, Audio,Text
MixEval-X: Any-to-Any Evaluations from Real-World Data Mixtures	MixEval-X	2024-10-17	Huggingface	Video,Image, Audio,Text

Star History

Similar repos

BradyFU/Awesome-Multimodal-Large-Language-Models

:sparkles::sparkles:Latest Advances on Multimodal Large Language Models

18,043

friedrichor/Awesome-Multimodal-Papers

A curated list of awesome Multimodal studies.

347

showlab/Awesome-Unified-Multimodal-Models

📖 This is a repository for organizing papers, codes and other resources related to unified…

834

lwpyh/Awesome-MLLM-Reasoning-Collection

A collection of multimodal reasoning papers, codes, datasets, benchmarks and resources.

OpenEnvision/Awesome-Multimodal-Modeling

Awesome Multimodal Modeling [Covers MLLM, UMM, and NMM]

547

Yuan-ManX/ai-multimodal-timeline

Here we will track the latest AI Multimodal Models, including Multimodal Foundation Models, LLM,…

YingqingHe/Awesome-LLMs-meet-Multimodal-Generation

🔥🔥🔥 A curated list of papers on LLMs-based multimodal generation (image, video, 3D and audio).

552

ATH-MaaS/Awesome-Unified-Multimodal-Models

Awesome Unified Multimodal Models

1,322