👀 Browse and Concentrate: Comprehending Multimodal Content via prior-LLM Context Fusion (ACL '24 Oral)

🌐 Homepage | 📖 arXiv | 🤗 Models

This is the pretraining data for Brote models. Please refer to the github repo for details and instructions.

info

• NOTE! Please use the newest version (updated on 20 Dec 2024). The previous version is deprecated.

• There are 8 parquet files compressed using gzip format.

  • stage1_gpt_v0_vqa_imgpath.parquet.gzip
  • stage1_gpt_v0_vcr_imgpath.parquet.gzip
  • stage1_gpt_v0_nlvr2_imgpath.parquet.gzip
  • stage1_gpt_v02_stvqa_imgpath_resample.parquet.gzip
  • stage1_gpt_v02_iconqa_imgpath_resample.parquet.gzip
  • stage1_gpt_v02_vsr_imgpath_resample.parquet.gzip
  • stage1_gpt_v02_ivqa_imgpath.parquet.gzip
  • stage1_gpt_v02_CGD_imgpath.parquet.gzip

• The parquet data contains 7 keys:

  • 'input_text': input text
  • 'input_image': list of paths of input images in 'input_text'
  • 'output_text': output text (or the target answer to generate, this is the target for fine-tuning)
  • 'ori_dataset': original dataset
  • 'model': model used for generating the Context-Enhanced captions
  • 'gpt_caption': the Context-Enhanced caption (this is the target for pre-training)
  • 'blip2_caption': captions generated by blip2, this used for mix_blip2 training (see the training scripts and data preparing scripts.)

• About images:

  • A dir 'images' is placed together with the data files, containing 8 compressed image dirs.

Reference

📑 If you find our data helpful to your research, please consider citing:

@inproceedings{
wang2024browse,
title={Browse and Concentrate: Comprehending Multimodal Content via Prior-{LLM} Context Fusion},
author={Wang, Ziyue and Chen, Chi and Zhu, Yiqi and Luo, Fuwen and Li, Peng and Yan, Ming and Zhang, Ji and Huang, Fei and Sun, Maosong and Liu, Yang},
booktitle={The 62nd Annual Meeting of the Association for Computational Linguistics},
year={2024},
}