Howto-Interlink7M

📙 Overview

Howto-Interlink7M presents a unique interleaved video-text dataset, carefully derived from the raw video content of Howto100M.

In the creation of this dataset, we turn a long video into a vision-text interleaved documents by BLIP2 (Img Captioner), GRIT (Img Detector), Whisper (ASR). Similar to VLog.

Then, we employed the GPT-4 for an extensive 7 million high-quality pretraining data.

During this process, we meticulously filtered out clips containing sensitive or low-quality content.

📊 Statistics

The statictis are listed below:

🎨 Visualization

Please see Youtube for more examples.

🏋️ Training

Please refer to code cosmo for training details.

Download Source Video

1. Download the README and All-in-One zip file:

On the official website HowTo100M, locate the download links for the README and the All-in-One zip file. Extract the contents of the All-in-One zip file:

2. Inside the extracted folder, you should find the HowTo100M_v1.csv file.

3. In the CSV file, you will find a column named "video_id" which contains unique identifiers for each video.

You can use youtube-dl or similar tools to download the videos using the video IDs listed in the CSV file.

🎓 Citation

@article{wang2024cosmo,
  title={COSMO: Contrastive Streamlined Multimodal Model with Interleaved Pre-Training},
  author={Wang, Alex Jinpeng and Li, Linjie and Lin, Kevin Qinghong and Wang Jianfeng and Lin, Kevin and Yang, Zhengyuan  and Wang, Lijuan and Shou, Mike Zheng},
  journal={arXiv preprint arXiv:2401.00849},
  year={2024}
}