Daily-Omni: Towards Audio-Visual Reasoning with Temporal Alignment across Modalities

This is the official repository of Daily-Omni: Towards Audio-Visual Reasoning with Temporal Alignment across Modalities

News

• [2026-07-26] Added AGIBOT X-Lab WITA-Omni Preview, Qwen3.5-Omni-Plus, Gemini 3.1 Pro Preview, and Doubao Seed 2.0 Lite to the audio-visual leaderboard. We thank AGIBOT for providing the evaluation results for Gemini, Doubao, and Qwen, and congratulate AGIBOT X-Lab WITA-Omni Preview on achieving the best performance on Daily-Omni!
• [2026-04-27] Added NVIDIA Nemotron 3 Nano Omni 30B A3B to the leaderboard as the current top evaluated open-weight omni-modal model.
• [2026-03-11] Updated the arXiv paper and refreshed the Leaderboard with new results.

Leaderboard

Performance comparison of MLLMs on Daily-Omni. Random guess accuracy is 25%.

Abbreviations:

• AV Align: audio-visual alignment
• Comp.: comparative
• Ctx. Und.: context understanding
• Evt. Seq.: event sequence
• Infer.: inference
• Reas.: reasoning
• 30s / 60s: duration-based subsets

Closed-source models are marked with (Closed) and open-source models with (Open).

Omni-Modal Language Models (With Visual and Audio)

^† Qwen3.5-Omni-Plus returned valid answers for 1,175 of 1,197 samples. The remaining 22 samples could not be evaluated after repeated API attempts because of request-size limits (13) and content filtering (9). Its scores use the valid API responses in each subset as the denominator.

Omni-Modal Language Models (Visual Only)

Omni-Modal Language Models (Audio Only)

Visual Language Models (Visual Only)

Audio Language Models (Audio Only)

Textual Language Models (Without Visual and Audio)

Requirements

To install requirements:

pip install -r requirements.txt

To test our benchmark, you should download Videos.tar and 'qa.json' from Huggingface to this directory and extract the Videos/ folder and qa.json to this directory.

QA Generation

📋 We provide script to reproduce our Daily-Omni QA Generation pipeline. Run the following command to generate QA pairs. To run the script, first you should revise the config.py file to set the parameters:

• Set the api keys, base_urls and model_name. You can create api keys from the following links: Gemini, OpenAI, Deepseek, Aliyun

• Set BASE_DIR and CSV_PATH to the Video Folder you want to annotate and the path to the csv file that records the videos. You can use example_videos and example_metadata.csv as templetes.

• Set MAX_WORKERS_PROCESSES to the number of processes you want to use to run the pipeline. You can set execution_mode in run_pipeline.py to choose the execution mode if your api key has parallel requests limitation.

• Set run_pipeline_flags in run_pipeline.py to choose which part of the pipeline you want to run.

Test the pipeline with:

python run_pipeline.py

Test Daily-Omni

Model with api

📋To test the benchmark on third party models(Gemini, GPT-4o, Deepseek) with api and reproduce the results, you can use the script provided in test_model_api/

python test_model_api/test_model.py --model <model_name> --mode <Execution_mode> --max_items <Maximum number of QA items to process (for testing)>

You can check the model options in test_model_api/test_config.py

Model running locally

📋To test the benchmark on third party models(Qwen2.5-Omni, Qwen2.5-VL, VideoLLaMA2, Ola, Unified-IO 2) with local machines, check the code in test_model/

All local test_model/*/testmodel.py scripts now use a unified modality argument:

--input_mode {all,visual,audio}

The default is --input_mode all.

• all: video + audio
• visual: video only
• audio: audio only

--use_audio_in_video has been removed. For scripts that save per-item JSONL results, raw model output is saved by default.

Qwen2.5-Omni

You should install the dependencies with instructions from the official Qwen2.5-Omni repo

Run the test script with the following command:

python test_model/Qwen2.5-Omni/testmodel.py --video_base_dir VIDEO_BASE_DIR --json_file_path JSON_FILE_PATH --model_name_or_path MODEL_NAME_OR_PATH --processor_name_or_path PROCESSOR_NAME_OR_PATH --input_mode all

You can switch modality with --input_mode all|visual|audio.

Qwen2.5-VL

You should install the dependencies with instructions from the official Qwen2.5-VL repo

Run the test script with the following command:

python test_model/Qwen2.5-VL/testmodel.py --video_base_dir VIDEO_BASE_DIR --json_file_path JSON_FILE_PATH --model_name_or_path MODEL_NAME_OR_PATH --processor_name_or_path PROCESSOR_NAME_OR_PATH --input_mode all

Qwen2.5-VL is visual-only. --input_mode all will fall back to visual, and audio is not supported.

Qwen3-VL

You should install the dependencies with instructions from the official Qwen3-VL repo

Run the test script with the following command:

python test_model/Qwen3-VL/testmodel.py --video_base_dir VIDEO_BASE_DIR --json_file_path JSON_FILE_PATH --model_name_or_path MODEL_NAME_OR_PATH --input_mode all --batch_size 8

Qwen3-VL is visual-only. --input_mode all will fall back to visual, and audio is not supported.

VideoLLaMA2

You should install the dependencies with instructions from the official VideoLLaMA2 repo

Run the test script with the following command:

python test_model/VideoLLaMA2-av/testmodel.py --video_base_dir VIDEO_BASE_DIR --json_file_path JSON_FILE_PATH --input_mode all

Unified-IO 2

You should install the dependencies with instructions from the official Unified-IO 2 repo

Run the test script with the following command:

python test_model/unified-io-2.pytorch/testmodel.py --video_base_dir VIDEO_BASE_DIR --json_file_path JSON_FILE_PATH --model MODEL_NAME --input_mode all

unified-io-2.pytorch currently only supports --input_mode all.

Ola

You should install the dependencies with instructions from the official Ola repo

Run the test script with the following command:

python test_model/Ola/inference/testmodel.py --video_base_dir VIDEO_BASE_DIR --json_file_path JSON_FILE_PATH

Ola currently uses its own script interface and is not aligned to the unified --input_mode CLI above.

Subsampling stability test

Use run_subsampling_stability.sh to run subsampling stability evaluation for the current model set.

./run_subsampling_stability.sh

What it does:

• Runs per-item evaluation first and saves *_items.jsonl for each model.
• Samples 20,40,60,80,100% of questions by default.
• Uses all QA belonging to the sampled videos to compute accuracy.
• Repeats the experiment R times and reports mean ± std and median [p5, p95].

Default model set:

• Qwen/Qwen2.5-Omni-7B-Instruct
• Qwen/Qwen3-Omni-30B-A3B-Instruct
• gemini-2.5-flash
• gemini-2.5-flash-lite

Useful environment variables:

• QWEN_INPUT_MODE / QWEN25_INPUT_MODE: modality, default all
• STABILITY_REPEATS: repeat count, default 200
• SUBSAMPLE_PCTS: comma-separated percentages, default 20,40,60,80,100
• INCLUDE_QWEN_THINKING=1: include Qwen3-Omni-30B-A3B-Thinking
• GEMINI_API_KEY_1 and GEMINI_API_KEY_2: Gemini API keys

Outputs are written to runs/subsampling_stability_<timestamp>/, including per-item JSONL, stability_repeat_results.csv, stability_summary.csv, and stability_summary.txt.

Test Daily-Omni Agent

We provide a script to test Daily-Omni Agent on Daily-Omni benchmark. For efficiency, we used the API provided by Bailian Aliyun for Qwen2.5-VL and Qwen2.5, while Qwen2-Audio was deployed locally. According to the official documentation, qwen2.5-vl-7b-instruct and qwen2.5-14b-instruct provided by Bailian Aliyun are identical to their open-source counterparts. Our code implements direct passing of local_video_path to the Qwen2.5-VL API. However, this functionality might require you to contact Aliyun customer service to enable. If direct path input is not activated, you can alternatively pass a list of video frames, though this may result in suboptimal performance.

To run the agent, you need to setup a new environment for Qwen2-Audio according to the instructions in the official repository.

Then, launch the Qwen2-Audio server locally with running:

python baseline/qwen_audio.py

Segment the video and audio clips:

python baseline/segment_av.py

Run Daily-Omni Agent on Daily-Omni benchmark with the following command:

python baseline/base_model.py

This script will automatically evaluate the performance of the model on the Daily-Omni benchmark.

Citation

@misc{zhou2026dailyomniaudiovisualreasoningtemporal,
      title={Daily-Omni: Towards Audio-Visual Reasoning with Temporal Alignment across Modalities}, 
      author={Ziwei Zhou and Rui Wang and Zuxuan Wu and Yu-Gang Jiang},
      year={2026},
      eprint={2505.17862},
      archivePrefix={arXiv},
      primaryClass={cs.AI},
      url={https://arxiv.org/abs/2505.17862}, 
}