Overview: Align-Anything Dataset

A Comprehensive All-Modality Alignment Dataset with Fine-grained Preference Annotations and Language Feedback.

🏠 Homepage | 🤗 Align-Anything Dataset | 🤗 T2T_Instruction-tuning Dataset | 🤗 TI2T_Instruction-tuning Dataset | 👍 Our Official Code Repo

Our world is inherently multimodal. Humans perceive the world through multiple senses, and Language Models should operate similarly. However, the development of Current Multi-Modality Foundation Models faces limitations due to the availability and diversity of data across different modalities. Specifically, the challenges include:

1. Imbalance in modality data: While there is abundant data for vision tasks, data for other modalities such as video and audio is relatively scarce, and there is a lack of interconnected data across different modalities.
2. Limited multi-modality training data: The majority of existing datasets focus on modality-specific question-answer tasks, while there is a lack of specialized datasets to enhance multi-modality models' Instruction-Following capabilities.

To address these challenges, we propose Align-Anything 200K, which features:

• All-modality tasks: Incorporating tasks that cover all major modalities.
• Fine-grained preference: Capturing nuanced user preferences across tasks.
• Language feedback: Supporting critique and refinement through natural language.
• Cross-modality QA pairs: Enabling richer interactions between different modalities.

Please cite the repo if you find the data or code in this repo useful 😊

@inproceedings{ji2024align,
  title={Align Anything: Training All-Modality Models to Follow Instructions with Language Feedback},
  author={Jiaming Ji and Jiayi Zhou and Hantao Lou and Boyuan Chen and Donghai Hong and Xuyao Wang and Wenqi Chen and Kaile Wang and Rui Pan and Jiahao Li and Mohan Wang and Josef Dai and Tianyi Qiu and Hua Xu and Dong Li and Weipeng Chen and Jun Song and Bo Zheng and Yaodong Yang},
  year={2024},
  url={https://arxiv.org/abs/2412.15838}
}

Summary

Our current open-source datasets

You can click the links in Modality Type for more details.

Prompt Distribution

Usage

from datasets import load_dataset

# text-to-text 
train_dataset = load_dataset('PKU-Alignment/align-anything',name='text-to-text')['train']
val_dataset = load_dataset('PKU-Alignment/align-anything',name='text-to-text')['val']

# text-image-to-text
train_dataset = load_dataset('PKU-Alignment/align-anything',name='text-image-to-text')['train']
val_dataset = load_dataset('PKU-Alignment/align-anything', name='text-image-to-text')['val']

# text-image-to-text-expert
train_dataset = load_dataset('PKU-Alignment/align-anything',name='text-image-to-text-expert')['train']
val_dataset = load_dataset('PKU-Alignment/align-anything', name='text-image-to-text-expert')['val']

# text-to-image
train_dataset = load_dataset('PKU-Alignment/align-anything',name='text-to-image')['train']
val_dataset = load_dataset('PKU-Alignment/align-anything', name='text-to-image')['val']

# text-audio-to-text
train_dataset = load_dataset('PKU-Alignment/align-anything',name='text-audio-to-text')['train']
val_dataset = load_dataset('PKU-Alignment/align-anything', name='text-audio-to-text')['val']

# text-to-audio
train_dataset = load_dataset('PKU-Alignment/align-anything',name='text-to-audio')['train']

# Due to the specificity of video files, we recommend using the `wget` command to download the video-based dataset directly.
# text-video-to-text:
wget https://huggingface.co/datasets/PKU-Alignment/align-anything/resolve/main/text-video-to-text/train_10k.json
wget https://huggingface.co/datasets/PKU-Alignment/align-anything/resolve/main/text-video-to-text/videos.tar.gz

# text-to-video:
wget https://huggingface.co/datasets/PKU-Alignment/align-anything/resolve/main/text-to-video/9k_train.json
wget https://huggingface.co/datasets/PKU-Alignment/align-anything/resolve/main/text-to-video/videos.tar.gz0
wget https://huggingface.co/datasets/PKU-Alignment/align-anything/resolve/main/text-to-video/videos.tar.gz1
wget https://huggingface.co/datasets/PKU-Alignment/align-anything/resolve/main/text-to-video/videos.tar.gz2
cat videos.tar.gz* | tar -xzvf

# text-image-to-text-image
# Load the ti2ti dataset with the `wget` command:
wget https://huggingface.co/datasets/PKU-Alignment/align-anything/resolve/main/text-image-to-text-image/train_27k.json
wget https://huggingface.co/datasets/PKU-Alignment/align-anything/resolve/main/text-image-to-text-image/images.tar.gz

# text-image-to-text instruction-following
train_dataset = load_dataset('PKU-Alignment/Align-Anything-TI2T-Instruction-100K',split='train')

# text-to-text instruction-following
dataset = load_dataset("PKU-Alignment/Align-Anything-Instruction-100K",split='train')

1. Highlights

Unlike existing datasets, which focus on individual modalities and vary in quality, Align-Anything offers consistent, high-quality data that encompasses any modality (e.g., text, image, video and audio) in mixed inputs and outputs. It provides detailed human preference annotations along with fine-grained language feedback for critique and refinement, enabling comprehensive evaluation and improvement across modalities.

1.1 All-Modality Tasks

We present the combination of our Align-Anything, divided into three parts:

• Any-to-Any represents the bidirectional conversion of any type of input-output modality, such as text, video, audio and images.
• Any-to-Text represents the transition from non-textual inputs—such as image, video, and audio—into textual output.
• Text-to-Any represents the setting that text inputs are to be converted into any other modalities.

1.2 Fine-Grained Preference

How to Define a High-Quality Image? Assessing the quality of rich multimodal data is challenging with binary preferences on individual metrics.

To address this, we have designed Fine-Grained All-Modality Constitutions to assist in annotating fine-grained preferences. These constitutions are composed of two main parts:

1. General fine-grained metrics across modalities, such as instruction-following, objective rules, clarity & aesthetics, information richness, and safety.
2. Modality-specific constitutions: For instance, for the video modality, we designed metrics such as temporal consistency, content coherence, and motion naturalness.

You can explore each modality’s subset dataset to view its fine-grained constitutions and definitions in detail.

According to the Fine-Grained All-Modality Constitutions, we utilized GPT-4o, Gemini-1.5-Pro, and Human Crowds to annotate data, resulting in comprehensive fine-grained annotations across all modalities.

1.3 Language Feedback

Multimodal data requires fine-grained annotations for better optimization. To guide the optimization process more effectively, multimodal data requires more fine-grained annotations. We propose a unified alignment method across all modalities by utilizing language feedback. Specifically, we provide critique and refinement feedback on each dimension as well as overall preferences for every data point. This feedback can be incorporated into your training process to enhance the performance of multimodal models.

1.4 Cross-Modality QA Pairs

Handling the interactions between different modalities is crucial for Multimodal Foundation Models. To address this, we have also labeled Any-to-Any Cross-Modality data, which allows for comprehensive interactions across modalities.

This dataset will be available soon...

2. Annotation Pipeline

We demonstrate a multi-step process for refining AI responses based on multi-modal prompts. Raw prompts are refined based on specific modality and task, and then used to generate responses from various sources. Finally, we used the closed-source SOTA model and humans to perform cross-modality fine-grained annotation and language feedback to obtain the final dataset.

2.1 Collect Q-A Pairs

We start by designing specialized features tailored to various modalities. Based on specific modality tasks and their corresponding feature designs, we design Fine-Grained All-Modality Constitutions, according to which we refine the original prompts, which may initially be suboptimal, to create the final versions. We then collect responses from multiple sources, including self-constructed methods, the invocation of open-source and closed-source models, and human-generated answers.

2.2 Fine-grained Annotation

We conduct fine-grained preference annotations on the collected question-answer pairs. The annotations are sourced from both GPT-4, Gemini-1.5-Pro and human annotators. This annotation process covers a diverse range of dimensions, such as instruction-following, objective rules, aesthetics, information richness and safety, each with corresponding preferences and scoring criteria.

2.3 Language Feedback

Finally, we provide language feedback on the responses. This involves determining the scope of critique, executing the critique, and providing refinement suggestions within the pipeline. This process captures both direct preferences for each modality and language-based feedback, ensuring a comprehensive evaluation and enhancement of the responses.

3. Datasets Comparison

Note
Existing preference datasets are limited in scope and quality, focusing on specific modalities and lacking comprehensive annotations. In contrast, Align-Anything offers high-quality data across all modalities, with detailed human preference annotations and language feedback for critique and refinement. This comprehensive approach ensures a consistent evaluation and improvement of responses across modalities.

Preference Annotation Methods in the table consist of three parts, namely Methods (A | S | F) in the above table.

• A refers to the annotation source, which indicates how preferences are determined within the dataset. "Manual" denotes human annotation or manually constructed preferences, "Synthetic" refers to preferences generated or annotated by models like GPT-4V or other systems, and "Combined" refers to datasets aggregated from multiple sources.
• S represents the composition of preference signals, which may include scoring, ranking, and reasoning. In some cases, preferences are constructed by refining, correcting, or corrupting responses to form the desired preference pairs.
• F indicates whether the dataset provides fine-grained feedback at a more detailed level within those preference dimensions.

Dimensions indicate the primary preference challenges the dataset aims to address.

We compare the existing multimodal preference datasets, as shown in the table above. This comparison highlights the feedback diversity in our Align-Anything, which addresses the limitations of existing preference datasets, particularly following the expansion into multiple modalities.

4. Human Agreement Analysis

We analyze the human agreement on the preference scores and the percentage of agreement on the preference scores. Our results show that the human agreement on the preference scores is high, indicating the reliability of the preference annotations. The percentage of agreement on the preference scores is also high, demonstrating the consistency of the preference annotations.

5. Citation

Please cite our work if you use the data or model in your paper.

@misc{align_anything,
  author = {PKU-Alignment Team},
  title = {Align Anything: training all modality models to follow instructions with unified language feedback},
  year = {2024},
  publisher = {GitHub},
  journal = {GitHub repository},
  howpublished = {\url{https://github.com/PKU-Alignment/align-anything}},
}