Spatial Temporal Video Understanding in MLLMs using Grounded Multimodal VideoQAs

• GLUS_FT contains files created by fine-tuning GLUS¹.
• Semi-automatic data generation and annotation pipeline was created using:
  • RoadSocial² dataset for obtaining Videos and QAs.
  • GPT to extract information from QAs to formulate RVOS prompts, and to score video clips for keyframes extraction.
  • Sa2VA³ as the RVOS model for segmenting objects in the video.
  • SAM2⁴ for tracking the objects segmented using Sa2VA throughout the video.
  • VideLLaMA3⁵ fine-tuned on the RoadSocial dataset as the Video LLM to extract keyframes from long-form videos.
• data_ann_tool.py contains code for the custom data annotation to verify the correctness of generated video object segmentation tracks
• Some benchmarking and fine-tuning code used later in the project is not added here as the project was discontinued after concurrent works solved the problem before our submission

---

Footnotes

1. Lin, Lang, et al. "Glus: Global-local reasoning unified into a single large language model for video segmentation." Proceedings of the Computer Vision and Pattern Recognition Conference. 2025.\ ↩

2. Parikh, Chirag, et al. "Roadsocial: A diverse videoqa dataset and benchmark for road event understanding from social video narratives." Proceedings of the Computer Vision and Pattern Recognition Conference. 2025.\ ↩

3. Yuan, Haobo, et al. "Sa2va: Marrying sam2 with llava for dense grounded understanding of images and videos." arXiv preprint arXiv:2501.04001 (2025).\ ↩

4. Ravi, Nikhila, et al. "Sam 2: Segment anything in images and videos." International Conference on Learning Representations. Vol. 2025. 2025.\ ↩

5. Zhang, Boqiang, et al. "Videollama 3: Frontier multimodal foundation models for image and video understanding." arXiv preprint arXiv:2501.13106 (2025). ↩