A Comprehensive Multi-modal Interaction Benchmark in Streaming Video Contexts

Updates

• 2026-07-14 Big Congrats to MOSS-VL-Realtime for achieving 66.0 on PA@OmniMMI!
• 2026-04-10 We appreciate the community bringing this to our attention. In Table 3, the second-to-last row accidentally replicates SG’s results for M4 on MD 1st–3rd. Please use the avg for reference.
• 2025-12-18 Big Congrats to Seed1.8 for achieving 53.0 on PA@OmniMMI!
• 2025-04-02 First Release Open-Omni-Nexus. a fully open-source implementation of a GPT-4o-like speech-to-speech video understanding model.
• 2025-04-01 Meet M4. M4 enables multiplexed modeling capabilities for a video language model at minimal cost.
• 2025-04-01 First Release OmniMMI. We're excited to share the first version of OmniMMI.

Table of Contents

• OmniMMI
  • Introduction
  • Statistics
  • Data
• Evaluation
  • Installation
  • Usage
• Leaderboard

OmniMMI

Introduction

we introduce OmniMMI, a comprehensive multi-modal interaction benchmark tailored for OmniLLMs in streaming video contexts. OmniMMI encompasses over 1,121 interactive videos and 2,290 questions, addressing two critical yet underexplored challenges in existing video benchmarks: streaming video understanding and proactive reasoning, across six distinct subtasks.

• Streaming Temporal State Awareness. Streaming video understanding must build an understanding w.r.t. the current and historical temporal state incrementally, without accessing the future context. This contrasts with traditional MLLM that can leverage the entire multi-modal contexts, posing challenges in our distinguished tasks of action prediction (AP), state grounding (SG) and multi-turn dependencies (MD).
• Proactive Reasoning and Turn-Taking. Generating responses proactively and appropriately anticipating the turn-taking time spot w.r.t. user's intentions and dynamic contexts is a crucial feature for general interactive agents. This typically requires models to identify speakers (SI), distinguish between noise or legitimate query (PT), and proactively initiate a response (PA).

Statistics

Data

You can download the OmniMMI from huggingface, containing both json and raw videos.

videohallucer_datasets                
    ├── clips/ # video clips for assessing offline models
    ├── videos/
    ├── action_prediction.json
    ├── dynamic_state_grounding.json
    ├── multiturn_dependency_reasoning.json
    ├── proactive_alerting.json
    ├── proactive_turntaking.json
    └── speaker_identification.json
  

We offer a selection of data samples (multiturn dependency reasoning) from our dataset for further elucidation:

[
    {
        "video": "jGM8EImhpXg.mp4",
        "qa": [
            {
                "question": "What is the task of the instruction video?",
                "answer": "Cook Omelet",
                "timestamp": "0.0 -- 36.0"
            },
            {
                "question": "What's the first step of ##ANSWER##?",
                "answer": "Pouring the egg into the pot",
                "timestamp": "36.0 -- 40.0"
            },
            {
                "question": "What do you do after the step from ##ANSWER##?",
                "answer": "fry eggs",
                "timestamp": "41.0 -- 70.0"
            }
        ]
    },
...
]

Evaluation

If you want to upload results from your models, feel free to submit a PR following one of these baselines, or send an email to me (flagwyx@gmail.com) to update your results on our page.

Installation

Available Baselines

Large Video Language Model

• VideoChatGPT-7B
• VideoChat2-7B
• Video-LLaVA-7B
• LLaMA-VID-7B/13B
• MiniGPT4-Video-7B
• PLLaVA-7B/13B/34B
• LLaVA-NeXT-Video-DPO-DPO-7B/34B
• ShareGPT4Video-8B
• LongVA-7B
• LongVILA-8B
• LongLLaVA-9B
• VideoLLM-online-8B
• VideoLLaMB-7B
• InternLM-XComposer-2.5-OmniLive-1.5B
• Gemini-1.5-pro
• GPT4O (Azure)

Large Omni Language Model

• VideoLLaMA2-7B
• VITA-7B
• MiniOmni2-0.5B
• M4-7B

For detailed instructions on installation and checkpoints of these baselines, please consult the INSTALLATION guide.

Note: To evaluate audio-input models, we use a TTS model to automatically convert text queries into audio. Please install ChatTTS beforehand.

Usage

debug inference pipeline

cd baselines
python ../model_testing_zoo.py --model_name Gemini-1.5-pro # ["VideoChatGPT", "VideoChat2", "VideoLLaVA", "LLaMA-VID", "PLLaVA", "PLLaVA-13B", "PLLaVA-34B", "LLaVA-NeXT-Video", "LLaVA-NeXT-Video-34B", "LongVA", "LongVILA", "LongLLaVA", "VideoLLaMB", "VideoOnline", "VideoLLaMBOneline", "Gemini-1.5-pro", "GPT4O"])

1. evaluate on OmniMMI at one step

cd baselines
bash run_all.sh
bash eval_all.sh

2. evaluate step by step

for each evaluation script for specific subtask, e.g. run_ap.sh

2.1 set the model and conda environment

model_names=("LongVA" "PLLaVA-13B")
environments=("llongva" "pllava")

There are two groups of models

(i) common models

"MiniGPT4-Video" "VideoChatGPT" "VideoLLaVA" "VideoChat2" "LLaMA-VID" "PLLaVA" "LLaVA-NeXT-Video" "ShareGPT4Video" "PLLaVA-13B" "PLLaVA-34B" "LLaVA-NeXT-Video-34B" "LongVA" "LongVILA" "LongLLaVA" "VideoLLaMB" "InternLMXCO" "InterSuit" "InterSuitAV"

(ii) online models

"VideoOnline" "VideoLLaMBOnline" "InterSuitOnline" "InterSuitOnlineAV"

2.2 set the conda environment path

source ~/scratch/anaconda3/bin/activate

2.3 run the evaluation script for each subtask

cd baselines
bash run_ap.sh # action prediction
bash eval_ap.sh

Leaderboard

Leaderboard of Large Video Language Model

Leaderboard of Large Omni Language Model

Please check the detailed leaderboard in our page

Citation

If you find our work helpful, please consider citing it.

@article{omnimmi,
    title={OmniMMI: A Comprehensive Multi-modal Interaction Benchmark in Streaming Video Contexts},
    author={Wang, Yuxuan and Wang, Yueqian and Chen, Bo and Wu, Tong and Zhao, Dongyan and Zheng, Zilong},
    journal={arxiv},
    year={2025}
}