2026.06.11 Videos re-encoded to H265, maintaining consistent evaluation scores. Fixed 2 incorrect MP4s & 3 mismatched URLs. Original data preserved in the original branch.
2026.05.22 Task types are now available for Q1-Q3 in coherence (logic) groups.
This repository contains annotation data for "Video-MME-v2: Towards the Next Stage in Benchmarks for Comprehensive Video Understanding". It mainly consists of three parts: videos/, test.parquet, and subtitle.zip.
videos/ contains 800 1080p MP4 files, organized sequentially into 40 zip archives. For example, 001.mp4 to 020.mp4 are stored in 001.zip.
test.parquet contains 3200 QA instances, with each video paired with 4 questions. Each instance includes the question, options, answer, and auxiliary metadata such as the video id and task type.
subtitle.zip contains 800 JSONL files, each corresponding to a unique video id, with word-level entries and timestamps.
In 2024, our Video-MME benchmark became a standard evaluation set for frontier models like Gemini and GPT. However, as model capabilities rapidly evolve, scores on existing benchmarks are saturating, yet a clear gap remains between leaderboard performance and actual user experience. This indicates that current evaluation paradigms fail to capture true video understanding abilities. To address this, we spent a year redesigning the evaluation system from first principles and now introduce Video-MME v2—a progressive and robust benchmark designed to drive the next generation of video understanding models.
Dataset Size
The dataset consists of 800 videos and 3,200 QA pairs, with each video associated with four MCQ-based questions.
Multi-level Evaluation Hierarchy
Group-based Evaluation Strategy
Video Sources
All videos are collected from YouTube. Over 80% were published in 2025 or later, with nearly 40% published after October 2025.
Video Categories
The dataset includes four top-level domains, further divided into 31 fine-grained subcategories.
Metrics
A non-linear scoring mechanism is applied to all question groups, and a first error truncation mechanism is used for reasoning coherence groups.
💡 Why this example matters? This video QA group demonstrates our Reasoning Coherence evaluation strategy and Multi-level Hierarchy. To answer the final state correctly, a model must successfully track the object backwards through temporal swaps. If a model guesses the initial state correctly but fails the intermediate swaps, our first error truncation mechanism will accurately penalize it for flawed reasoning.
👆 Click the cover image to view the demo video.
Q1: Did the ball exist underneath any of the shells?
A. No.
B. Yes. ✅
C. Cannot be determined.
Q2: Underneath which shell was the ball located at the end?
A. There is no ball under any shell.
B. The third shell.
C. The sixth shell.
D. The second shell.
E. The seventh shell.
F. The fifth shell.
G. The fourth shell. ✅
H. The first shell.
Q3: The host performed a total of two shell swaps (defining a single swap as an instance where all shells return to an approximately straight line). Underneath which shell was the ball located after the first swap?
A. There is no ball under any shell.
B. The seventh shell.
C. The fourth shell. ✅
D. The fifth shell.
E. The sixth shell.
F. The second shell.
G. The third shell.
H. The first shell.
Q4: The host performed a total of two shell swaps (defining a single swap as an instance where all shells return to an approximately straight line). Underneath which shell was the ball located initially?
A. The seventh shell.
B. The fourth shell.
C. The fifth shell.
D. The third shell. ✅
E. The second shell.
F. There is no ball under any shell.
G. The first shell.
H. The sixth shell.
2026.06.11 Videos re-encoded to H265, maintaining consistent evaluation scores. Fixed 2 incorrect MP4s & 3 mismatched URLs. Original data preserved in the original branch.
2026.05.22 Task types are now available for Q1-Q3 in coherence (logic) groups.
This repository contains annotation data for "Video-MME-v2: Towards the Next Stage in Benchmarks for Comprehensive Video Understanding". It mainly consists of three parts: videos/, test.parquet, and subtitle.zip.
videos/ contains 800 1080p MP4 files, organized sequentially into 40 zip archives. For example, 001.mp4 to 020.mp4 are stored in 001.zip.
test.parquet contains 3200 QA instances, with each video paired with 4 questions. Each instance includes the question, options, answer, and auxiliary metadata such as the video id and task type.
subtitle.zip contains 800 JSONL files, each corresponding to a unique video id, with word-level entries and timestamps.
In 2024, our Video-MME benchmark became a standard evaluation set for frontier models like Gemini and GPT. However, as model capabilities rapidly evolve, scores on existing benchmarks are saturating, yet a clear gap remains between leaderboard performance and actual user experience. This indicates that current evaluation paradigms fail to capture true video understanding abilities. To address this, we spent a year redesigning the evaluation system from first principles and now introduce Video-MME v2—a progressive and robust benchmark designed to drive the next generation of video understanding models.
Dataset Size
The dataset consists of 800 videos and 3,200 QA pairs, with each video associated with four MCQ-based questions.
Multi-level Evaluation Hierarchy
Group-based Evaluation Strategy
Video Sources
All videos are collected from YouTube. Over 80% were published in 2025 or later, with nearly 40% published after October 2025.
Video Categories
The dataset includes four top-level domains, further divided into 31 fine-grained subcategories.
Metrics
A non-linear scoring mechanism is applied to all question groups, and a first error truncation mechanism is used for reasoning coherence groups.
💡 Why this example matters? This video QA group demonstrates our Reasoning Coherence evaluation strategy and Multi-level Hierarchy. To answer the final state correctly, a model must successfully track the object backwards through temporal swaps. If a model guesses the initial state correctly but fails the intermediate swaps, our first error truncation mechanism will accurately penalize it for flawed reasoning.
👆 Click the cover image to view the demo video.
Q1: Did the ball exist underneath any of the shells?
A. No.
B. Yes. ✅
C. Cannot be determined.
Q2: Underneath which shell was the ball located at the end?
A. There is no ball under any shell.
B. The third shell.
C. The sixth shell.
D. The second shell.
E. The seventh shell.
F. The fifth shell.
G. The fourth shell. ✅
H. The first shell.
Q3: The host performed a total of two shell swaps (defining a single swap as an instance where all shells return to an approximately straight line). Underneath which shell was the ball located after the first swap?
A. There is no ball under any shell.
B. The seventh shell.
C. The fourth shell. ✅
D. The fifth shell.
E. The sixth shell.
F. The second shell.
G. The third shell.
H. The first shell.
Q4: The host performed a total of two shell swaps (defining a single swap as an instance where all shells return to an approximately straight line). Underneath which shell was the ball located initially?
A. The seventh shell.
B. The fourth shell.
C. The fifth shell.
D. The third shell. ✅
E. The second shell.
F. There is no ball under any shell.
G. The first shell.
H. The sixth shell.
