👁️ Semi-Automatic Video Annotation Pipeline

---

📝 Description

Video-ChatGPT introduces the VideoInstruct100K dataset, which employs a semi-automatic annotation pipeline to generate 75K instruction-tuning QA pairs. To address the limitations of this annotation process, we present VCG+112K dataset developed through an improved annotation pipeline. Our approach improves the accuracy and quality of instruction tuning pairs by improving keyframe extraction, leveraging SoTA large multimodal models (LMMs) for detailed descriptions, and refining the instruction generation strategy.

💻 Download

To get started, follow these steps:

git lfs install
git clone https://huggingface.co/MBZUAI/video_annotation_pipeline

📚 Additional Resources

• Paper: ArXiv.
• GitHub Repository: For training and updates: GitHub - GLaMM.
• HuggingFace Collection: For downloading the pretrained checkpoints, VCGBench-Diverse Benchmarks and Training data, visit HuggingFace Collection - VideoGPT+.

📜 Citations and Acknowledgments

  @article{Maaz2024VideoGPT+,
      title={VideoGPT+: Integrating Image and Video Encoders for Enhanced Video Understanding},
      author={Maaz, Muhammad and Rasheed, Hanoona and Khan, Salman and Khan, Fahad Shahbaz},
      journal={arxiv},
      year={2024},
      url={https://arxiv.org/abs/2406.09418}
  }