MiraData: A Large-Scale Video Dataset with Long Durations and Structured Captions

Xuan Ju^1*, Yiming Gao^1*, Zhaoyang Zhang^1*#, Ziyang Yuan¹, Xintao Wang¹, Ailing Zeng, Yu Xiong, Qiang Xu, Ying Shan^1
1ARC Lab, Tencent PCG ²The Chinese University of Hong Kong ^*Equal Contribution ^#Project Lead

Introduction

Video datasets play a crucial role in video generation such as Sora. However, existing text-video datasets often fall short when it comes to handling long video sequences and capturing shot transitions. To address these limitations, we introduce MiraData, a video dataset designed specifically for long video generation tasks. Moreover, to better assess temporal consistency and motion intensity in video generation, we introduce MiraBench, which enhances existing benchmarks by adding 3D consistency and tracking-based motion strength metrics. You can find more details in our research papaer.

Dataset

Meta Files

We release four versions of MiraData, containing 330K, 93K, 42K, 9K data.

The meta file for this version of MiraData is provided here. Additionally, for a better and quicker understanding of our meta file composition, we randomly sample a set of 100 video clips, which can be accessed here. The meta file contains the following index information:

• clip_id: video clip index, which is composed of {download_id}.{clip_id}
• source: video download source and category
• video_url: video source url
• video_id: video id in the source website
• width: video width
• height: video height
• fps: video fps used for extracting frame
• seconds: duration of the video clip
• timestamp: clip start and end timestamp in source video (used for cutting the video clip from its source video)
• frame_number: frame number of the video clip
• framestamp: clip start and end frame in source video
• file_path: file path for storing the video clip
• short_caption: a short overall caption
• dense_caption: a dense overall caption
• background_caption: caption of the video background
• main_object_caption: caption of the main object in video
• style_caption: caption of the video style
• camera_caption: caption of the camera move

Download

To download the videos and split the videos into clips, you can use the following scripts:

python download_data.py --meta_csv {meta file} --download_start_id {the start of download id} --download_end_id {the end of download id} --raw_video_save_dir {the path of saving raw videos} --clip_video_save_dir {the path of saving cutted video}

^{We will remove the video samples from our dataset / Github / project webpage as long as you need it. Please contact to us for the request.}

Citation

If you find this project useful for your research, please cite our paper. :blush:

@misc{ju2024miradatalargescalevideodataset,
      title={MiraData: A Large-Scale Video Dataset with Long Durations and Structured Captions}, 
      author={Xuan Ju and Yiming Gao and Zhaoyang Zhang and Ziyang Yuan and Xintao Wang and Ailing Zeng and Yu Xiong and Qiang Xu and Ying Shan},
      year={2024},
      eprint={2407.06358},
      archivePrefix={arXiv},
      primaryClass={cs.CV},
      url={https://arxiv.org/abs/2407.06358}, 
}

Contact Information

For any inquiries, please email mira-x@googlegroups.com.