Inst-IT Dataset: An Instruction Tuning Dataset with Multi-level Fine-Grained Annotations

introduced in the paper Inst-IT: Boosting Multimodal Instance Understanding via Explicit Visual Prompt Instruction Tuning

🌐 Homepage | Code | 🤗 Paper | 📖 arXiv

Inst-IT Dataset Overview

We create a large-scale instruction tuning dataset, the Inst-it Dataset. To the best of our knowledge, this is the first dataset that provides fine-grained annotations centric on specific instances. Inst-it Dataset contains 21k videos and 51k images (we treat images as static, single-frame videos). In total, Inst-it Dataset includes :

• 21k videos
• 51k images
• 21k video-level descriptions
• 207k frame-level descriptions (51k images, 156k video frames) (each frame-level description includes captions of 1)individual instances, 2)the entire image, and 3)the temporal changes.)
• 335k open-ended QA pairs

We visualize the data structure in the figure below, and you can view a more detailed data sample [here].

Download Inst-IT Dataset

To download the Inst-IT Dataset, you can run the following scripts:

pip install -U "huggingface_hub[cli]"
huggingface-cli download --repo-type dataset --resume-download Inst-IT/Inst-IT-Dataset --local-dir Inst-IT-Dataset
cd Inst-IT-Dataset && unzip images_vpt.zip && unzip videos_vpt.zip && unzip images_raw.zip && unzip videos_raw.zip

Then, you will get a folder structure as follows:

Inst-IT-Dataset
|-- README.md 
|-- images_vpt # images with set-of-marks visual prompts.
|-- images_raw # images without set-of-marks visual prompts.
|-- videos_vpt # videos with set-of-marks visual prompts.
|-- videos_raw # videos without set-of-marks visual prompts.
|-- inst_it_dataset_image_51k.json # the annotation file of 51k images
`-- inst_it_dataset_video_21k.json # the annotation file of 21k videos

Note!!!We are working on releasing the SFT data in LLaVA format, as in LLaVA-Instruct-150K, please stay tuned! If you have any questions or suggestions, feel free to contact us :)

The annotations in inst_it_dataset_video_21k.json follow the structure as below:

[
    {
        "video_id": int,
        "frame_level_caption": (annotation for each frames within this video)
          [
              {
                  "timestamp": int, (indicate the timestamp of this frame in the video, e.g. <1>)
                  "frame_name": string, (the image filename of this frame)
                  "instance_level": (caption for each instance within this frame)
                    {
                        "1": "caption for instance 1",
                        (more instance level captions ...)
                    },
                  "image_level": string, (caption for the entire frame)
                  "temporal_change": string (caption for the temporal changes relative to the previous frame)
              },
              (more frame level captions ...)
          ],
        "question_answer_pairs": (open ended question answer pairs)
          [
             {
                "question": "the question",
                "answer": "the corresponding answer"
              },
             (more question answer pairs ...)
          ],
        "video_level_caption": string, (a dense caption for the entire video, encompassing all frames)
        "video_path": string (the path to where this video is stored)
    },
    (more annotations for other videos ...)
]

The annotations in inst_it_dataset_image_51k.json follow the structure as below:

[
    {
        "image_id": int,
        "instance_level_caption": (caption for each instance within this image)
          {
              "1": "caption for instance 1",
              (more instance level captions ...)
          },
        "image_level_caption": string, (caption for the entire image)
        "image_path": string (the path to where this image is stored)
    },
    (more annotations for other images ...)
]

Enhance LMMs with Inst-IT Dataset

Based on the Inst-IT Dataset, we propose a continuous instruction-tuning recipe to mix instance understanding datasets with generic instruction-tuning data effectively. By adding this small amount of data, the enhanced models demonstrate strong performance among various benchmarks and our [Inst-IT Bench].

The checkpoints of our model weights can be found:

• results on image benchmarks

• results on video benchmarks

Contact

Feel free to contact us if you have any questions or suggestions

• Email (Wujian Peng): wjpeng24@m.fudan.edu.cn
• Email (Lingchen Meng): lcmeng20@fudan.edu.cn

Citation

If you find our work helpful, please consider citing our paper ✒️ and like our dataset ❤️ :

 @article{peng2024boosting,
   title={Inst-IT: Boosting Multimodal Instance Understanding via Explicit Visual Prompt Instruction Tuning},
   author={Peng, Wujian and Meng, Lingchen and Chen, Yitong and Xie, Yiweng and Liu, Yang and Gui, Tao and Hang, Xu and Qiu, Xipeng and Wu, Zuxuan and Jiang, Yu-Gang},
   journal={arXiv preprint arXiv:2412.03565},
   year={2024}
 }