This project aims to collect and collate various datasets for multimodal large model training, including but not limited to pre-training data, instruction fine-tuning data, and In-Context learning data.
78
15 commits
updated May 7, 2025
README
Awesome-MLLM-Datasets
🚀🚀🚀This project aims to collect and collate various datasets for multimodal large model training, including but not limited to pre-training data, instruction fine-tuning data, and In-Context learning data.
💡💡💡The goal of the project is to provide researchers with a comprehensive repository of resources to support their ability to more easily access high-quality datasets when developing and optimizing multimodal AI systems.
Table of Contents
- Datasets of Pre-Training
- Datasets of Multimodal Instruction Tuning
- Datasets of In-Context Learning
- Datasets of Multimodal Chain-of-Thought
- Datasets of Multimodal RLHF
- Benchmarks for Evaluation
Datasets of Pre-Training
Datasets of Multimodal Instruction Tuning
Datasets of In-Context Learning
| Name | Paper | Link | Notes |
|---|---|---|---|
| MIC | MMICL: Empowering Vision-language Model with Multi-Modal In-Context Learning | Link | A manually constructed instruction tuning dataset including interleaved text-image inputs, inter-related multiple image inputs, and multimodal in-context learning inputs. |
| MIMIC-IT | MIMIC-IT: Multi-Modal In-Context Instruction Tuning | Link | Multimodal in-context instruction dataset |
Datasets of Multimodal Chain-of-Thought
| Name | Paper | Link | Notes |
|---|---|---|---|
| EMER | Explainable Multimodal Emotion Reasoning | Link | A benchmark dataset for explainable emotion reasoning task |
| EgoCOT | EmbodiedGPT: Vision-Language Pre-Training via Embodied Chain of Thought | Link | Large-scale embodied planning dataset |
| VIP | Let’s Think Frame by Frame: Evaluating Video Chain of Thought with Video Infilling and Prediction | - | An inference-time dataset that can be used to evaluate VideoCOT |
| ScienceQA | Learn to Explain: Multimodal Reasoning via Thought Chains for Science Question Answering | Link | Large-scale multi-choice dataset, featuring multimodal science questions and diverse domains |
Datasets of Multimodal RLHF
| Name | I->O | Method | #.Instance | Paper | Link |
|---|---|---|---|---|---|
| VLFeedback | I+T->T | Auto | 80K | Silkie: Preference Distillation for Large Visual Language Models | Link |
| LLaVA-RLHF | I+T->T | Manu | 10K | Aligning large multimodal models with factually augmented rlhf | Link |
| DRESS's IT | I+T->T | Auto+Manu | - | DRESS : Instructing Large Vision-Language Models to Align and Interact with Humans via Natural Language Feedback | Link |
| RLHF-V's IT | I+T->T | Manu | 1.4K | RLHF-V: Towards Trustworthy MLLMs via Behavior Alignment from Fine-grained Correctional Human Feedback | Link |
| RTVLM | I+T->T | Auto+Manu | 5K | Red Teaming Visual Language Models | Link |
| VLGuard's IT | I+T->T | Auto | 3K | Safety Fine-Tuning at (Almost) No Cost: A Baseline for Vision Large Language Models | Link |
| MMViG | I+T->T | Manu | 16K | ViGoR: Improving Visual Grounding of Large Vision Language Models with Fine-Grained Reward Modeling | - |
Benchmarks for Evaluation
Similar repos
This project aims to collect and collate various datasets for multimodal large model training, including but not limited to pre-training data, instruction fine-tuning data, and In-Context learning data.
78
15 commits
updated May 7, 2025
README
Awesome-MLLM-Datasets
🚀🚀🚀This project aims to collect and collate various datasets for multimodal large model training, including but not limited to pre-training data, instruction fine-tuning data, and In-Context learning data.
💡💡💡The goal of the project is to provide researchers with a comprehensive repository of resources to support their ability to more easily access high-quality datasets when developing and optimizing multimodal AI systems.
Table of Contents
- Datasets of Pre-Training
- Datasets of Multimodal Instruction Tuning
- Datasets of In-Context Learning
- Datasets of Multimodal Chain-of-Thought
- Datasets of Multimodal RLHF
- Benchmarks for Evaluation
Datasets of Pre-Training
Datasets of Multimodal Instruction Tuning
Datasets of In-Context Learning
| Name | Paper | Link | Notes |
|---|---|---|---|
| MIC | MMICL: Empowering Vision-language Model with Multi-Modal In-Context Learning | Link | A manually constructed instruction tuning dataset including interleaved text-image inputs, inter-related multiple image inputs, and multimodal in-context learning inputs. |
| MIMIC-IT | MIMIC-IT: Multi-Modal In-Context Instruction Tuning | Link | Multimodal in-context instruction dataset |
Datasets of Multimodal Chain-of-Thought
| Name | Paper | Link | Notes |
|---|---|---|---|
| EMER | Explainable Multimodal Emotion Reasoning | Link | A benchmark dataset for explainable emotion reasoning task |
| EgoCOT | EmbodiedGPT: Vision-Language Pre-Training via Embodied Chain of Thought | Link | Large-scale embodied planning dataset |
| VIP | Let’s Think Frame by Frame: Evaluating Video Chain of Thought with Video Infilling and Prediction | - | An inference-time dataset that can be used to evaluate VideoCOT |
| ScienceQA | Learn to Explain: Multimodal Reasoning via Thought Chains for Science Question Answering | Link | Large-scale multi-choice dataset, featuring multimodal science questions and diverse domains |
Datasets of Multimodal RLHF
| Name | I->O | Method | #.Instance | Paper | Link |
|---|---|---|---|---|---|
| VLFeedback | I+T->T | Auto | 80K | Silkie: Preference Distillation for Large Visual Language Models | Link |
| LLaVA-RLHF | I+T->T | Manu | 10K | Aligning large multimodal models with factually augmented rlhf | Link |
| DRESS's IT | I+T->T | Auto+Manu | - | DRESS : Instructing Large Vision-Language Models to Align and Interact with Humans via Natural Language Feedback | Link |
| RLHF-V's IT | I+T->T | Manu | 1.4K | RLHF-V: Towards Trustworthy MLLMs via Behavior Alignment from Fine-grained Correctional Human Feedback | Link |
| RTVLM | I+T->T | Auto+Manu | 5K | Red Teaming Visual Language Models | Link |
| VLGuard's IT | I+T->T | Auto | 3K | Safety Fine-Tuning at (Almost) No Cost: A Baseline for Vision Large Language Models | Link |
| MMViG | I+T->T | Manu | 16K | ViGoR: Improving Visual Grounding of Large Vision Language Models with Fine-Grained Reward Modeling | - |