Awesome Unified Understanding and Generation

This is a repository for organizing unified understanding and generation works. We have organized the existing works in terms of paradigms. Specifically, some works focus on the research of unified tokenizers. For worksfocusing on unified model and framework, we classified them according to text generation methods and image generation methods. For example, "AR + Masked AR" means that the text uses the AR method, while image tokens adopt the Masked AR (discrete diffusion) method. Methods belong to "AR + Masked AR + Additional Diffusion (Two Model)" additionally use a diffusion network to decode image tokens on the basis mentioned above. Furthermore, "Seperate Understanding and Generation Transformers" indicates that the model decouples understanding and generation into two transformers. When performing image generation, the understanding transformer serves to encode semantic-rich query embeddings as conditions for generation.

If you have any suggestions, please feel free to open issues or contact us.

:star: If you find this repo useful, please star it!

1. Unified Tokenizer

• TokLIP: Marry Visual Tokens to CLIP for Multimodal Comprehension and Generation (May. 2025, arXiv)

• UniTok: A Unified Tokenizer for Visual Generation and Understanding (Feb. 2025, arXiv)

• UniToken: Harmonizing Multimodal Understanding and Generation through Unified Visual Encoding (Apr. 2025, arXiv)

• TokenFlow: Unified Image Tokenizer for Multimodal Understanding and Generation (Dec. 2024, arXiv)

• MUSE-VL: Modeling Unified VLM through Semantic Discrete Encoding (Nov. 2024, arXiv)

• Making LLaMA SEE and Draw with SEED Tokenizer (Oct. 2023, ICLR)

• SEED-X: Multimodal Models with Unified Multi-granularity Comprehension and Generation (Apr. 2024, arXiv)

• LaVIT: Unified Language-Vision Pretraining in LLM with Dynamic Discrete Visual Tokenization (Sep. 2023, ICLR)

• Planting a SEED of Vision in Large Language Model (Jul. 2023, arXiv)

2. AR + AR (One Model)

• Ovis2.5 Technical Report (Aug. 2025, arXiv)

• UGen: Unified Autoregressive Multimodal Model with Progressive Vocabulary Learning (Mar. 2025, arXiv)

• OmniMamba: Efficient and Unified Multimodal Understanding and Generation via State Space Models (Mar. 2025, arXiv)

• Liquid: Language Models are Scalable Multi-modal Generators (Dec. 2024, arXiv)

• SynerGen-VL: Towards Synergistic Image Understanding and Generation with Vision Experts and Token Folding (Dec. 2024, arXiv)

• Janus: Decoupling Visual Encoding for Unified Multimodal Understanding and Generation (Nov. 2024, arXiv)

• JetFormer: An Autoregressive Generative Model of Raw Images and Text (Nov. 2024, arXiv)

• Emu3: Next-Token Prediction is All You Need (Sep. 2024, arXiv)

• ANOLE: An Open, Autoregressive, Native Large Multimodal Models for Interleaved Image-Text Generation (Jul. 2024, arXiv)

• Chameleon: Mixed-Modal Early-Fusion Foundation Models (May 2024, arXiv)

• World Model on Million-Length Video And Language With Blockwise RingAttention (Feb. 2024, arXiv)

• AnyGPT: Unified Multimodal LLM with Discrete Sequence Modeling (Feb. 2024, arXiv)

• Video-LaVIT: Unified Video-Language Pre-training with Decoupled Visual-Motional Tokenization (Feb. 2024, arXiv)

3. AR + AR + Additional Diffusion (Two Model)

• Ovis-U1 Technical Report (June. 2025, arXiv)

• ILLUME+: Illuminating Unified MLLM with Dual Visual Tokenization and Diffusion Refinement (Apr. 2025, arXiv)

• MIO: A Foundation Model on Multimodal Tokens (Sep. 2024, arXiv)

• Orthus: Autoregressive Interleaved Image-Text Generation with Modality-Specific Heads (Nov. 2024, arXiv)

• PUMA: Empowering Unified MLLM with Multi-granular Visual Generation (Oct. 2024, arXiv)

• VILA-U: a Unified Foundation Model Integrating Visual Understanding and Generation (Sep. 2024, arXiv)

• X-VILA: Cross-Modality Alignment for Large Language Model (May. 2024, arXiv)

• Unified-IO 2: Scaling Autoregressive Multimodal Models with Vision, Language, Audio, and Action (Dec. 2023, arXiv)

• VL-GPT: A Generative Pre-trained Transformer for Vision and Language Understanding and Generation (Dec. 2023, arXiv)

• NExT-GPT: Any-to-Any Multimodal LLM (Sep. 2023, ICML)

• Emu2: Generative Multimodal Models are In-Context Learners (Jul. 2023, CVPR)

• Emu: Generative Pretraining in Multimodality (Jul. 2023, ICLR)

4. AR + Masked AR (One Model)

• UniGen: Enhanced Training & Test-Time Strategies for Unified Multimodal Understanding and Generation (May. 2025, arXiv)

• Show-o: One Single Transformer to Unify Multimodal Understanding and Generation (Aug. 2024, arXiv)

• Harmonizing Visual Representations for Unified Multimodal Understanding and Generation (Mar. 2025, arXiv)

5. AR + Masked AR + Additional Diffusion (Two Model)

• MMAR: Towards Lossless Multi-Modal Auto-Regressive Probabilistic Modeling (Oct. 2024, arXiv)

6. AR + Full Attention + Additional Diffusion (Two Model)

• Show-o2: Improved Native Unified Multimodal Models (May. 2025, arXiv)

• LlamaFusion: Adapting Pretrained Language Models for Multimodal Generation (Dec. 2024, arXiv)

7. AR + Continuous Diffusion (One Model)

• Emerging Properties in Unified Multimodal Pretraining (May. 2025, arXiv)

• MINT: Multi-modal Chain of Thought in Unified Generative Models for Enhanced Image Generation (Mar. 2025, arXiv)

• JanusFlow: Harmonizing Autoregression and Rectified Flow for Unified Multimodal Understanding and Generation (Nov. 2024, arXiv)

• OmniGen: Unified Image Generation (Sep. 2024, arXiv)

• MonoFormer: One Transformer for Both Diffusion and Autoregression (Sep. 2024, arXiv)

• Transfusion: Predict the Next Token and Diffuse Images with One Multi-Modal Model (Aug. 2024, arXiv)

8. Diffusion

• Mmada: Multimodal large diffusion language models (May. 2025, arXiv)

• MMGen: Unified Multi-modal Image Generation and Understanding in One Go (Mar. 2025, arXiv)

• Dual Diffusion for Unified Image Generation and Understanding (Jan. 2025, arXiv)

• OmniFlow: Any-to-Any Generation with Multi-Modal Rectified Flows (Dec. 2024, arXiv)

• CoDi: Any-to-Any Generation via Composable Diffusion (May. 2023, NeurIPS)

9. Seperate Understanding and Generation Transformers (Two Model)

• OmniGen2: Exploration to Advanced Multimodal Generationn (June. 2025, arXiv)

• Ming-Lite-Uni: Advancements in Unified Architecture for Natural Multimodal Interaction (May. 2025, arXiv)

• BLIP3-o: A Family of Fully Open Unified Multimodal Models—Architecture, Training and Dataset (May. 2025, arXiv)

• Transfer between Modalities with MetaQueries (Apr. 2025, arXiv)

• Nexus-Gen: A Unified Model for Image Understanding, Generation, and Editing (Apr. 2025, arXiv)

• MetaMorph: Multimodal Understanding and Generation via Instruction Tuning (Dec. 2024, arXiv)

• MM-Interleaved: Interleaved Image-Text Generative Modeling via Multi-modal Feature Synchronizer (Jan. 2024, arXiv)

• DreamLLM: Synergistic Multimodal Comprehension and Creation (Dec. 2023, ICLR)

Acknowledgements

This template is provided by Awesome-Unified-Multimodal-Models.