For each benchmark below, the nonblind split contains the questions that cannot be answered without the video:
three models (Qwen2.5-VL-7B-Instruct, Qwen3-VL-8B-Instruct, InternVL3.5-8B) were run with 4 black frames instead of
the video, and every question that at least two of them still answered correctly was removed. These are the pools
from which Meta-Video was sampled.
Rows keep the original columns of the source dataset (same names and values), so an lmms-eval task can be pointed at
this repo (dataset_path: soumyasj/video-benchmarks-nonblind, dataset_name: <config>, test_split: nonblind).
Added columns: black_qwen2_5_vl_7b, black_qwen3_vl_8b, black_internvl3_5_8b (black-frame score of each model,
0/1) and blind_correct_models (number of models with score >= 0.5).
speedup = original / nonblind: evaluating only the non-blind split costs that many times fewer forward passes than the full set (per-item cost is unchanged, so wall-clock scales the same way).
Videos are not included; use the video files of the original benchmarks.
| config | original | nonblind | removed | speedup |
|---|---|---|---|---|
videomme | 2700 | 1601 | 41% | 1.69x |
longvideobench_val_v | 1337 | 773 | 42% | 1.73x |
lvbench | 1549 | 1281 | 17% | 1.21x |
video_mmmu_perception | 300 | 206 | 31% | 1.46x |
video_mmmu_comprehension | 300 | 221 | 26% | 1.36x |
video_mmmu_adaptation | 300 | 226 | 25% | 1.33x |
mmvu_val | 1000 | 644 | 36% | 1.55x |
scivideobench | 1000 | 824 | 18% | 1.21x |
tempcompass_multi_choice | 1580 | 946 | 40% | 1.67x |
mvbench_action_antonym | 200 | 181 | 10% | 1.10x |
mvbench_action_count | 200 | 114 | 43% | 1.75x |
mvbench_action_localization | 200 | 139 | 30% | 1.44x |
mvbench_action_prediction | 200 | 162 | 19% | 1.23x |
mvbench_action_sequence | 200 | 149 | 26% | 1.34x |
mvbench_character_order | 200 | 122 | 39% | 1.64x |
mvbench_counterfactual_inference | 200 | 136 | 32% | 1.47x |
mvbench_egocentric_navigation | 200 | 146 | 27% | 1.37x |
mvbench_episodic_reasoning | 200 | 112 | 44% | 1.79x |
mvbench_fine_grained_action | 200 | 159 | 20% | 1.26x |
mvbench_fine_grained_pose | 200 | 155 | 22% | 1.29x |
mvbench_moving_attribute | 200 | 136 | 32% | 1.47x |
mvbench_moving_count | 200 | 124 | 38% | 1.61x |
mvbench_moving_direction | 200 | 169 | 16% | 1.18x |
mvbench_object_existence | 200 | 104 | 48% | 1.92x |
mvbench_object_interaction | 200 | 161 | 20% | 1.24x |
mvbench_object_shuffle | 200 | 151 | 24% | 1.32x |
mvbench_scene_transition | 200 | 107 | 46% | 1.87x |
mvbench_state_change | 200 | 139 | 30% | 1.44x |
mvbench_unexpected_action | 200 | 76 | 62% | 2.63x |
| all | 14066 | 9464 | 33% | 1.49x |
Sources: lmms-lab/Video-MME, longvideobench/LongVideoBench, lmms-lab/LVBench, lmms-lab/VideoMMMU, lmms-lab/MMVU, groundmore/scivideobench, lmms-lab/TempCompass, OpenGVLab/MVBench. Each keeps its original license.
For each benchmark below, the nonblind split contains the questions that cannot be answered without the video:
three models (Qwen2.5-VL-7B-Instruct, Qwen3-VL-8B-Instruct, InternVL3.5-8B) were run with 4 black frames instead of
the video, and every question that at least two of them still answered correctly was removed. These are the pools
from which Meta-Video was sampled.
Rows keep the original columns of the source dataset (same names and values), so an lmms-eval task can be pointed at
this repo (dataset_path: soumyasj/video-benchmarks-nonblind, dataset_name: <config>, test_split: nonblind).
Added columns: black_qwen2_5_vl_7b, black_qwen3_vl_8b, black_internvl3_5_8b (black-frame score of each model,
0/1) and blind_correct_models (number of models with score >= 0.5).
speedup = original / nonblind: evaluating only the non-blind split costs that many times fewer forward passes than the full set (per-item cost is unchanged, so wall-clock scales the same way).
Videos are not included; use the video files of the original benchmarks.
| config | original | nonblind | removed | speedup |
|---|---|---|---|---|
videomme | 2700 | 1601 | 41% | 1.69x |
longvideobench_val_v | 1337 | 773 | 42% | 1.73x |
lvbench | 1549 | 1281 | 17% | 1.21x |
video_mmmu_perception | 300 | 206 | 31% | 1.46x |
video_mmmu_comprehension | 300 | 221 | 26% | 1.36x |
video_mmmu_adaptation | 300 | 226 | 25% | 1.33x |
mmvu_val | 1000 | 644 | 36% | 1.55x |
scivideobench | 1000 | 824 | 18% | 1.21x |
tempcompass_multi_choice | 1580 | 946 | 40% | 1.67x |
mvbench_action_antonym | 200 | 181 | 10% | 1.10x |
mvbench_action_count | 200 | 114 | 43% | 1.75x |
mvbench_action_localization | 200 | 139 | 30% | 1.44x |
mvbench_action_prediction | 200 | 162 | 19% | 1.23x |
mvbench_action_sequence | 200 | 149 | 26% | 1.34x |
mvbench_character_order | 200 | 122 | 39% | 1.64x |
mvbench_counterfactual_inference | 200 | 136 | 32% | 1.47x |
mvbench_egocentric_navigation | 200 | 146 | 27% | 1.37x |
mvbench_episodic_reasoning | 200 | 112 | 44% | 1.79x |
mvbench_fine_grained_action | 200 | 159 | 20% | 1.26x |
mvbench_fine_grained_pose | 200 | 155 | 22% | 1.29x |
mvbench_moving_attribute | 200 | 136 | 32% | 1.47x |
mvbench_moving_count | 200 | 124 | 38% | 1.61x |
mvbench_moving_direction | 200 | 169 | 16% | 1.18x |
mvbench_object_existence | 200 | 104 | 48% | 1.92x |
mvbench_object_interaction | 200 | 161 | 20% | 1.24x |
mvbench_object_shuffle | 200 | 151 | 24% | 1.32x |
mvbench_scene_transition | 200 | 107 | 46% | 1.87x |
mvbench_state_change | 200 | 139 | 30% | 1.44x |
mvbench_unexpected_action | 200 | 76 | 62% | 2.63x |
| all | 14066 | 9464 | 33% | 1.49x |
Sources: lmms-lab/Video-MME, longvideobench/LongVideoBench, lmms-lab/LVBench, lmms-lab/VideoMMMU, lmms-lab/MMVU, groundmore/scivideobench, lmms-lab/TempCompass, OpenGVLab/MVBench. Each keeps its original license.
