Dataset Card for LLaVA-OneVision

[2024-09-01]: Uploaded VisualWebInstruct(filtered), it's used in OneVision Stage

almost all subsets are uploaded with HF's required format and you can use the recommended interface to download them and follow our code below to convert them.

the subset of ureader_kg and ureader_qa are uploaded with the processed jsons and tar.gz of image folders. You may directly download them from the following url. https://huggingface.co/datasets/lmms-lab/LLaVA-OneVision-Data/tree/main/ureader_kg

In this dataset, we include the data splits used in the both final image stage and one-vision stage. For more details, please check our paper and our training doc.

Dataset Description

• Curated by: Bo Li, Kaichen Zhang, Hao Zhang, Yuanhan Zhang, Renrui Zhang, Feng Li, Dong Guo
• Language(s) (NLP): English, Chinese
• License: Apache License 2.0

Dataset Sources

• Dataset Collection: We include a few subsets from existing dataset collection Cambrian, Cauldron, UReader. Since we only used a few subsets from these datasets, and applied the cleaning and re-annotation process, we uploaded our processed version of these datasets into our own repository and thank the authors for providing the original datasets.
• Other Datasets: For rest single source dataset, such as AI2D, OKVQA, we cite and link the original sources in our paper.

Uses

This dataset is used for the training of the LLaVA-OneVision model. We only allow the use of this dataset for academic research and education purpose. For OpenAI GPT-4 generated data, we recommend the users to check the OpenAI Usage Policy.

Dataset Structure

We expalin the data composition for mid-stage and final-stage at our repo in training doc.

Statistics

We provide the statistics of the dataset in the following figures, and refer the audience to check our paper.

Code Guidance

To help audience to better understand our dataest, we upload them into Hugging Face Dataset compatible format. During LLaVA-OneVision training, we use the json and image/video folder to store the data.

the subset of ureader_kg and ureader_qa are uploaded with the processed jsons and tar.gz of image folders. You may directly download them from the following url. https://huggingface.co/datasets/lmms-lab/LLaVA-OneVision-Data/tree/main/ureader_kg

Here we provide the code guidance to convert the dataset into the format of LLaVA-OneVision, and conduct the training of the LLaVA-OneVision model with converted dataset.

import os
from datasets import load_dataset
from tqdm import tqdm
import json

data = load_dataset("lmms-lab/LLaVA-OneVision-Data", split="train")

image_folder = "<your_image_folder>"

converted_data = []

for da in tqdm(data):
    json_data = {}
    json_data["id"] = da["id"]
    if da["image"] is not None:
        json_data["image"] = f"{da['id']}.jpg"
        da["image"].save(os.path.join(image_folder, json_data["image"]))
    json_data["conversations"] = da["conversations"]
    converted_data.append(json_data)


with open("<your_json_file>.json", "w") as f:
    json.dump(converted_data, f, indent=4, ensure_ascii=False)

Citation

BibTeX:

[More Information Needed]

Glossary

The dataset collection process is conducted by all of the authors, we thank the Feng Li and Renrui Zhang for providing LLaVA-M4-Instruct Data and Yuanhan for providing the Video datasets.

After the dataset collection, the cleaning and re-annotation process, including final mixture of the dataset, is conducted by Bo Li and with the great help of Kaichen Zhang.

Dataset Card Authors

The dataset is curated by the following authors:

Bo Li, Kaichen Zhang, Hao Zhang, Yuanhan Zhang, Renrui Zhang, Feng Li

Dataset Card Contact

Bo Li: drluodian@gmail.com

Kaichen Zhang