CrossVid: A Comprehensive Benchmark for Evaluating Cross-Video Reasoning in Multimodal Large Language Models 🎬

---

🌟 Introduction

CrossVid is the first comprehensive benchmark for evaluating Cross-Video Reasoning (CVR) in Multimodal Large Language Models (MLLMs). Unlike existing benchmarks focusing on single-video analysis, CrossVid challenges models to simultaneously understand, aggregate, and compare information across multiple videos.

Key Highlights:

• 🎯 First systematic CVR benchmark with hierarchical task design
• 📊 9,015 QA pairs across 5,331 videos from 6 diverse datasets
• 🏗️ 10 specific tasks spanning 4 dimensions (Comparative, Temporal, Multi-View, Free-Form)
• 🌐 32 genres covering real-world scenarios
• ⏱️ Long-context: Average 770 seconds per query
• 📝 Multiple formats: Single-choice, multiple-choice, and open-ended questions

---

📢 News

• [2026-06] 🔥 CrossVid is used by Seed 2.0 and Seed 2.1 to test multi-video understanding capability.
• [2025-11] 🎉 CrossVid accepted by AAAI 2026!
• [2025-11] 📊 Dataset available on HuggingFace.
• [2025-11] 🔧 Evaluation code uploaded.

---

🎯 Benchmark Overview

Task Dimensions

📊 Comparative Analysis - Behavioral Understanding (BU), Narrative Comprehension (NC), Culinary Comparison (CC), and Procedural Eror Analysis (PEA)

⏱️ Temporal Understanding - Plot Inference (PI), Functional Step Alignment (FSA), Procedural Step Sequencing (PSS)

👁️ Multi-View Reasoning - Multi-view Spatial Reasoning (MSR) and Multi-view Object Counting (MOC)

✍️ Free-Form QA - Comparative Culinary QA (CCQA)

Data Sources & Statistics

Videos from 6 public datasets: Animal Kingdom 🦁 | MovieChat-1K 🎬 | YouCook2 👨‍🍳 | VisDrone 🚁 | Charades 🏠 | Assembly101 🔧.

We thank the creators of these valuable datasets for providing the foundational video resources.

---

📸 Examples

---

🏗️ Annotation Pipeline

Process: Frame Extraction (Qwen2.5-VL-72B) → QA Generation (DeepSeek-R1) → Manual Filtration → Refinement → Quality Control

---

🚀 Quick Start

We provide a evaluation script named by the task name that supports parallel inference using OpenAI-compatible APIs (e.g., vLLM, LMDeploy, or SGLang).

1. Preparation

Due to copyright restrictions, please download the Charades and Animal Kingdom datasets from their official repositories.
After downloading, merge all original videos and place them under:

videos/behavior

Before running the evaluation, download annotations and other videos from HuggingFace and clone this repository. Ensure your environment are set up correctly:

Directory Structure
Ensure your project directory looks like this:

CrossVid/
│── uav/                 # VisDrone
│   ├── bbox/
│   └── frames/
├── videos/              # Folder containing video files
│   ├── assembly/        # Assembly101
│   ├── behavior/        # Charades & Animal Kingdom
│   ├── cook/            # YouCook2
│   └── movie/           # MovieChat-1K
│── QA/                  # Folder containing QA JSON files
│   ├── BU.json
│   ├── CC.json
│   ├── CCQA.json
│   ├── ...
|── eval/                # The evaluation scripts
│   ├── utils/
│   ├── BU.py
│   ├── CC.py
│   ├── ...
│   ├── score_CCQA.py
└── README.md

Python environment
Install the following required packages:

pip install openai opencv-python decord numpy

Enter root directory:

cd CrossVid

2. Run Evaluation

To evaluate a task, run the evaluation script with the following command. The script will process videos, perform inference via the API, and automatically calculate the accuracy. For example, you can evaluate task BU via:

python eval/BU.py \
    --model "your-model-name" \
    --video_root "videos" \
    --QA_path "QA/BU.json" \
    --save_path "results/BU_result.json" \
    --port 8000 \
    --threads 20

3. Arguments

4. Output & Metrics

Upon completion, the script saves detailed results to the specified JSON file and prints the overall accuracy:

The performance of <model_name> on task BU is 0.654

5. Open-ended Evaluation

For open-ended tasks (e.g., CCQA), we employ an LLM-as-a-Judge approach to score responses based on Coverage and Correctness of key scoring points.

Remember to configure the API key/URL in eval/score_CCQA.py.

python eval/score_CCQA.py \
    --QA_path "QA/CCQA.json" \
    --answer_path "results/CCQA_result.json" \
    --save_path "results/CCQA_score.json"

---

📊 Leaderboard

Benchmark Results

The following table shows the performance of 22 evaluated MLLMs on CrossVid dataset, ranked by Overall Average (O.Avg) score.

Model Leaderboard

Note: An equal number of frames are sampled uniformly from each video and resized to 360px on the longer side.

---

Metrics Description

• O.Avg: Overall average accuracy across all ten tasks
• C.Avg: Average accuracy on Comparative Analysis tasks (BU, NC, CC, PEA)
• T.Avg: Average accuracy on Temporal Understanding tasks (PI, FSA, PSS)
• M.Avg: Average accuracy on Multi-view Reasoning tasks (MSR, MOC)
• CCQA: Comparative Culinary QA accuracy
• #Frames: Total number of input frames per query

Note: Bold numbers in each column indicate the best performance among models in that category.

---

📄 License & Contact

License:

• This repository (evaluation scripts): MIT License - see LICENSE
• QA annotations on HuggingFace: CC BY 4.0
• Source videos on HuggingFace: same licenses as their corresponding original datasets (Animal Kingdom, MovieChat-1K, YouCook2, VisDrone, Charades, Assembly101)

We do not own any copyrights of these videos. All video contents are from public datasets, and their copyrights belong to the original authors / dataset creators.

Contact:

• Email: chuntianli666666@gmail.com
• GitHub: CrossVid

Acknowledgements: Thanks to dataset authors and our expert annotators.

---

📝 Citation

If you find CrossVid useful for your research, please cite our paper:

@article{li2025crossvid,
  title={CrossVid: A Comprehensive Benchmark for Evaluating Cross-Video Reasoning in Multimodal Large Language Models},
  author={Li, Jingyao and Wang, Jingyun and Tan, Molin and Wang, Haochen and Yan, Cilin and Shi, Likun and Cai, Jiayin and Jiang, Xiaolong and Hu, Yao},
  journal={arXiv preprint arXiv:2511.12263},
  year={2025}
}

---