Dataset Card for LLaVA-Video-178K

Dataset Description

• Curated by: Yuanhan Zhang, Jinming Wu, Wei Li
• Language(s) (NLP): English, Chinese
• License: Apache License 2.0

Uses

This dataset is used for the training of the LLaVA-Video model. We only allow the use of this dataset for academic research and education purpose. For OpenAI GPT-4 generated data, we recommend the users to check the OpenAI Usage Policy.

Data Sources

For the training of LLaVA-Video, we utilized video-language data from five primary sources:

• LLaVA-Video-178K: This dataset includes 178,510 caption entries, 960,792 open-ended QA (question and answer) items, and 196,198 multiple-choice QA items. These data were newly annotated for this project.
  • We include this dataset in this repository: LLaVA-Video-178K/XXX_academic_v0_1 and LLaVA-Video-178K/XXX_youtube_v0_1.
• NeXT-QA: Comprises 17,090 open-ended QA items and 17,024 multiple-choice QA items.
  • We include this dataset in this repository: LLaVA-Video-178K/XXX_nextqa.
• ActivityNetQA: Includes 23,530 open-ended QA items,
  • We include this dataset in this repository: LLaVA-Video-178K/XXX_activitynetqa.
• PerceptionTest: Includes 1,803 open-ended QA items.
  • We include this dataset in this repository: LLaVA-Video-178K/XXX_perceptiontest.
• LLaVA-Hound: Contains 240,000 open-ended QA items and 15,000 caption entries.
  • The video data and annotations are available at the following URLs:
  • Video data: train_300k
  • Annotation data: LLaVA-Video-178K/llava_hound
  • loading function is specified here: function

The LLaVA-Video-178K dataset is the only contribution from this repository; we provide additional datasets for reproducing LLaVA-Video.

• Project Page: Project Page.
• Paper: For more details, please check our paper

Annotation Pipeline

The following directories are provided for generating captions and QA data:

• Captions: LLaVA-Video-178K/gpt4o_caption_prompt
• QA: LLaVA-Video-178K/gpt4o_qa_prompt

The subset used in the LLaVA-OneVision

We have included captions and open-ended questions in the 0_30_s_academic_v0_1 split, along with 240,000 open-ended QA items and 15,000 caption entries, as part of the video data in LLaVA-Hound for LLaVA-OneVision.

• 0_30_s_academic_v0_1 caption
• 0_30_s_academic_v0_1 open-ended QA
• LLaVA-Hound: Same as above.

Citation

@misc{zhang2024videoinstructiontuningsynthetic,
    title={Video Instruction Tuning With Synthetic Data}, 
    author={Yuanhan Zhang and Jinming Wu and Wei Li and Bo Li and Zejun Ma and Ziwei Liu and Chunyuan Li},
    year={2024},
    eprint={2410.02713},
    archivePrefix={arXiv},
    primaryClass={cs.CV},
    url={https://arxiv.org/abs/2410.02713}, 
}

Dataset Card Contact

Yuanhan Zhang

Jinming Wu

Wei Li