Datasets for training VARGPT

arxiv

Dataset Structure

1. stage1-pt: Contains 1.28M pre-training instruction fine-tuning dataset for VARGPT.

2. stage2-sft: Includes datasets for the second stage of VARGPT instruction fine-tuning:

   • stage2-sft/llava_v1_5_mix665k: Derived entirely from LLaVA-1.5 training data.
   • stage2-sft/llava_onevision_508k: Sampled from the LLaVA-onevision Dataset.
   • stage2-sft/ImageNet-Instruct-5k: Sampled from our stage3-sft/ImageNet-Instruct-130k dataset.

3. stage3-sft: Contains datasets for the third stage of VARGPT instruction fine-tuning:

   • stage3-sft/ImageNet-Instruct-130k
   • stage3-sft/ImageNet-Instruct-1270k

Image Data Download and Processing

To prepare the image data, download the following datasets:

1. ImageNet
2. COCO: train2017
3. GQA: images
4. OCR-VQA: images
5. TextVQA: train_val_images
6. VisualGenome: part1, part2
7. LLaVA-Onevision Dataset

Matching Instruction Fine-tuning Data Paths with Image Paths

For datasets ImageNet-Instruct-class, ImageNet-Instruct-130k, and ImageNet-Instruct-1270k, use the following Python script to match image paths with ImageNet image UIDs:

from tqdm import tqdm
import glob
import os
import json
import pandas as pd

def build_image_mapping(parquet_folder_path):
    image_mapping = {}
    label_mapping = {}
    parquet_files = glob.glob(os.path.join(parquet_folder_path, "train-*.parquet"))
    
    for file_idx, parquet_file in enumerate(tqdm(parquet_files)):
        file_path = os.path.join(parquet_folder_path, parquet_file)
        df = pd.read_parquet(file_path)
        
        for idx in range(len(df)):
            img_path = df.iloc[idx]['image']['path']
            image_filename = f"image_{file_idx}_{idx}.jpg"
            prefix = img_path.split('_')[0] + '_' + img_path.split('_')[1]
            image_mapping[prefix] = image_filename
            label_mapping[os.path.join("", image_filename)] = df.iloc[idx]['label']
    return image_mapping, label_mapping

image_mapping, label_mapping = build_image_mapping("ImageNet/data")

This script creates two mappings:

1. image_mapping: Maps the original image prefixes to new image filenames.
2. label_mapping: Associates the new image filenames with their corresponding labels.

Additional Notes

• Ensure all datasets are downloaded and stored in appropriate directories before running the script.
• The script assumes a specific structure for the ImageNet parquet files. Adjust the path and file naming convention if necessary.
• After running the script, you can use image_mapping and label_mapping for further data processing or model training tasks.

For any issues or further clarification, please refer to the project documentation or contact the maintainers.

Citation

To cite the datasets and model, please use the below:

This datasets are used in the paper VARGPT-v1.1: Improve Visual Autoregressive Large Unified Model via Iterative Instruction Tuning and Reinforcement Learning:

@misc{zhuang2025vargptunifiedunderstandinggeneration,
      title={VARGPT: Unified Understanding and Generation in a Visual Autoregressive Multimodal Large Language Model}, 
      author={Xianwei Zhuang and Yuxin Xie and Yufan Deng and Liming Liang and Jinghan Ru and Yuguo Yin and Yuexian Zou},
      year={2025},
      eprint={2501.12327},
      archivePrefix={arXiv},
      primaryClass={cs.CV},
      url={https://arxiv.org/abs/2501.12327}, 
}
@misc{zhuang2025vargptv11improvevisualautoregressive,
      title={VARGPT-v1.1: Improve Visual Autoregressive Large Unified Model via Iterative Instruction Tuning and Reinforcement Learning}, 
      author={Xianwei Zhuang and Yuxin Xie and Yufan Deng and Dongchao Yang and Liming Liang and Jinghan Ru and Yuguo Yin and Yuexian Zou},
      year={2025},
      eprint={2504.02949},
      archivePrefix={arXiv},
      primaryClass={cs.CV},
      url={https://arxiv.org/abs/2504.02949}, 
}