StreamingBench: Assessing the Gap for MLLMs to Achieve Streaming Video Understanding

StreamingBench evaluates Multimodal Large Language Models (MLLMs) in real-time, streaming video understanding tasks. 🌟

---

[NEW! 2025.05.15] 🔥: Seed1.5-VL achieved ALL model SOTA with a score of 82.80 on the Proactive Output.

[NEW! 2025.03.17] ⭐: ViSpeeker achieved Open-Source SOTA with a score of 61.60 on the Omni-Source Understanding.

[NEW! 2025.01.14] 🚀: MiniCPM-o 2.6 achieved Streaming SOTA with a score of 66.01 on the Overall benchmark.

[NEW! 2025.01.06] 🏆: Dispider achieved Streaming SOTA with a score of 53.12 on the Overall benchmark.

[NEW! 2024.12.09] 🎉: InternLM-XComposer2.5-OmniLive achieved 73.79 on Real-Time Visual Understanding.

---

🎞️ Overview

As MLLMs continue to advance, they remain largely focused on offline video comprehension, where all frames are pre-loaded before making queries. However, this is far from the human ability to process and respond to video streams in real-time, capturing the dynamic nature of multimedia content. To bridge this gap, StreamingBench introduces the first comprehensive benchmark for streaming video understanding in MLLMs.

Key Evaluation Aspects

• 🎯 Real-time Visual Understanding: Can the model process and respond to visual changes in real-time?
• 🔊 Omni-source Understanding: Does the model integrate visual and audio inputs synchronously in real-time video streams?
• 🎬 Contextual Understanding: Can the model comprehend the broader context within video streams?

Dataset Statistics

• 📊 900 diverse videos
• 📝 4,500 human-annotated QA pairs
• ⏱️ Five questions per video at different timestamps

🎬 Video Categories

🔍 Task Taxonomy

🔬 Experimental Results

Performance of Various MLLMs on StreamingBench

• All Context

• 60 seconds of context preceding the query time

• Comparison of Main Experiment vs. 60 Seconds of Video Context
• 

Performance of Different MLLMs on the Proactive Output Task

"≤ xs" means that the answer is considered correct if the actual output time is within x seconds of the ground truth.

📝 Citation

@article{lin2024streaming,
  title={StreamingBench: Assessing the Gap for MLLMs to Achieve Streaming Video Understanding},
  author={Junming Lin and Zheng Fang and Chi Chen and Zihao Wan and Fuwen Luo and Peng Li and Yang Liu and Maosong Sun},
  journal={arXiv preprint arXiv:2411.03628},
  year={2024}
}

https://arxiv.org/abs/2411.03628