Data-centric Multimodal LLM

Survey on data-centric multimodal large language models

Paper

List of Sources

Commonly-used datasets

Textual-Pretraining Datasets:

MM-Pretraining Datasets:

Common Textual SFT Datasets:

Domain Specific Textual SFT Datasets:

Multimodal SFT Datasets:

Data-centric pretraining

Domain mixture

1. Doremi: Optimizing data mixtures speeds up language model pretraining - paper
2. Data selection for language models via importance resampling - paper
3. Glam: Efficient scaling of language models with mixture-of-experts - paper
4. Videollm: Modeling video sequence with large language models - paper
5. Vast: A vision-audio-subtitle-text omni-modality foundation model and dataset - paper
6. Moviechat: From dense token to sparse memory for long video understanding - paper
7. Internvid: A large-scale video-text dataset for multimodal understanding and generation - paper
8. Youku-mplug: A 10 million large-scale chinese video-language dataset for pre-training and benchmarks - paper

Modality Mixture

1. MM1: Methods, Analysis & Insights from Multimodal LLM Pre-training - paper
2. From scarcity to efficiency: Improving clip training via visual-enriched captions - paper
3. Valor: Vision-audio-language omni-perception pretraining model and dataset - paper
4. AutoAD:Moviedescription in context - paper
5. Chat-UniVi: Unified Visual Representation Empowers Large Language Models with Image and Video Understanding - paper
6. VideoChat: Chat-Centric Video Understanding - paper
7. Mvbench: A comprehensive multi-modal video understanding benchmark - paper
8. LLaMA-VID: An image is worth 2 tokens in large language models - paper
9. Video-llava:Learningunitedvisualrepresentation by alignment before projection - paper
10. Valley: Video assistant with large language model enhanced ability - paper
11. Video-llama: An instruction-tuned audio-visual language model for video understanding - paper
12. Macaw-LLM: Multi-Modal Language Modeling with Image, Audio, Video, and Text Integration - paper
13. Audio-Visual LLM for Video Understanding - paper
14. Fine-grained Audio-Visual Joint Representations for Multimodal Large Language Models - paper

Quality Selection

1. DataComp: In search of the next generation of multimodal datasets - paper
2. Finetuned Multimodal Language Models Are High-Quality Image-Text Data Filters - paper
3. CiT: Curation in Training for Effective Vision-Language Data - paper
4. Sieve: Multimodal Dataset Pruning Using Image Captioning Models - paper
5. Variance Alignment Score: A Simple But Tough-to-Beat Data Selection Method for Multimodal Contrastive Learning - paper

Data-centric adaptation

Data-Centric Supervised Finetuning

1. Unnatural instructions: Tuning language models with (almost) no human labor - paper
2. Active Learning for Convolutional Neural Networks: A Core-Set Approach - paper
3. Moderate Coreset: A Universal Method of Data Selection for Real-world Data-efficient Deep Learning - paper
4. Similar: Submodular information measures based active learning in realistic scenarios - paper
5. Practical coreset constructions for machine learning - paper
6. Deep learning on a data diet: Finding important examples early in training - paper
7. A new active labeling method for deep learning - paper
8. Maybe only 0.5% data is needed: A preliminary exploration of low training data instruction tuning - paper
9. DEFT: Data Efficient Fine-Tuning for Pre-Trained Language Models via Unsupervised Core-Set Selection - paper
10. Beyond neural scaling laws: beating power law scaling via data pruning - paper
11. Mods: Model-oriented data selection for instruction tuning. - paper
12. DeBERTa: Decoding-enhanced BERT with Disentangled Attention - paper
13. Alpagasus: Training a better alpaca with fewer data - paper
14. Rethinking the Instruction Quality: LIFT is What You Need - paper
15. What Makes Good Data for Alignment? A Comprehensive Study of Automatic Data Selection in Instruction Tuning - paper
16. InsTag: Instruction Tagging for Analyzing Supervised Fine-tuning of Large Language Models - paper
17. SelectLLM: Can LLMs Select Important Instructions to Annotate? - paper
18. Improved Baselines with Visual Instruction Tuning - paper
19. NLU on Data Diets: Dynamic Data Subset Selection for NLP Classification Tasks - paper
20. LESS: Selecting Influential Data for Targeted Instruction Tuning - paper
21. From quantity to quality: Boosting llm performance with self-guided data selection for instruction tuning - paper
22. One shot learning as instruction data prospector for large language models - paper
23. Active Instruction Tuning: Improving Cross-Task Generalization by Training on Prompt Sensitive Tasks - paper
24. SelectIT: Selective Instruction Tuning for Large Language Models via Uncertainty-Aware Self-Reflection - paper

Data-Centric Human Preference Alignment

1. Training language models to follow instructions with human feedback - paper
2. LLaMA-VID: An image is worth 2 tokens in large language models - paper
3. Aligning large multimodal models with factually augmented rlhf - paper
4. Dress: Instructing large vision-language models to align and interact with humans via natural language feedback - paper

Evaluation

1. Gans trained by a two time-scale update rule converge to a local nash equilibrium - paper
2. Assessing generative models via precision and recall - paper
3. Unsupervised Quality Estimation for Neural Machine Translation - paper
4. Mixture models for diverse machine translation: Tricks of the trade - paper
5. The vendi score: A diversity evaluation metric for machine learning - paper
6. Cousins Of The Vendi Score: A Family Of Similarity-Based Diversity Metrics For Science And Machine Learning - paper
7. Navigating text-to-image customization: From lycoris fine-tuning to model evaluation - paper
8. TRUE: Re-evaluating factual consistency evaluation - paper
9. Object hallucination in image captioning - paper
10. Faithscore: Evaluating hallucinations in large vision-language models - paper
11. Deep coral: Correlation alignment for deep domain adaptation - paper
12. Transferability in deep learning: A survey - paper
13. Mauve scores for generative models: Theory and practice - paper
14. Translating Videos to Natural Language Using Deep Recurrent Neural Networks - paper

Evaluation Datasets: