Figure 1: Timeline of Publicly Available and Unavailable Unified Multimodal Models. The models are categorized by their release years, from 2023 to 2025. Models underlined in the diagram represent any-to-any multimodal models, capable of handling inputs or outputs beyond text and image, such as audio, video, and speech. The timeline highlights the rapid growth in this field.

🔥 We are hiring!

We are looking for both interns and full-time researchers to join our team, focusing on multimodal understanding, generation, reasoning, AI agents, and unified multimodal models. If you are interested in exploring these exciting areas, please reach out to us at qingguo.cqg@alibaba-inc.com.

👉 What is This Repo for?

This repository provides a comprehensive collection of resources related to unified multimodal models, featuring:

• A survey of advances, challenges, and timelines for unified models
• Categorized lists of diffusion-based, autoregressive (MLLM), and hybrid architectures for unified image–text understanding and generation
• Benchmarks for evaluating multimodal comprehension, image generation, and interleaved image–text tasks
• Representative datasets covering multimodal understanding, text-to-image synthesis, image editing, and interleaved interactions

Designed to help researchers and practitioners explore, compare, and build state-of-the-art unified multimodal systems.

Awesome Papers & Datasets

• Text-and-Image Unified Models
  • Diffusion
  • MLLM AR
  • MLLM AR-Diffusion
• Any-to-Any Multimodal models
• Benchmark for Evaluation
  • Benchmarks on Understanding Tasks
  • Benchmarks on Image Generation Tasks
  • Benchmarks on Interleaved Tasks
• Dataset
  • Multimodal Understanding
  • Text-to-Image
  • Image Editing
  • Interleaved Image-Text
  • Other Text+Image-to-Image
• Applications and Opportunities

Text-and-Image Unified Models

Figure 2: Classification of Unified Multimodal Understanding and Generation Models. The models are divided into three main categories based on their backbone architecture: Diffusion, MLLM (AR), and MLLM (AR + Diffusion). Each category is further subdivided according to the encoding strategy employed, including Pixel Encoding, Semantic Encoding, Learnable Query Encoding, and Hybrid Encoding. We illustrate the architectural variations within these categories and their corresponding encoder-decoder configurations.

Diffusion

MLLM AR

b-1: Pixel Encoding

b-2: Semantic Encoding

b-3: Learnable Query Encoding

b-4: Hybrid Encoding (Pseduo)

b-5: Hybrid Encoding (Joint)

MLLM AR-Diffusion

c-1: Pxiel Encoding

c-2: Hybrid Encoding (Pseduo)

Any-to-Any Multimodal models

Benchmark for Evaluation

Benchmarks on Understanding Tasks

Benchmarks on Image Generation Tasks

Benchmarks on Interleaved / Compositional / Other Tasks

Dataset

Multimodal Understanding

Text-to-Image

Image Editing

Interleaved Image-Text

Other Text-Image-to-Image

Applications and Opportunities

Citation

If you find this repo is helpful for your research, please cite our paper:

@article{zhang2025unified,
  title={Unified Multimodal Understanding and Generation Models: Advances, Challenges, and Opportunities},
  author={Zhang, Xinjie and Guo, Jintao and Zhao, Shanshan and Fu, Minghao and Duan, Lunhao and Hu, Jiakui and Chng, Yong Xien and Wang, Guo-Hua and Chen, Qing-Guo and Xu, Zhao and Luo, Weihua and Zhang, Kaifu},
  journal={arXiv preprint arXiv:2505.02567},
  year={2025}
}