Meta-Video is a 1,000-question evaluation set for video understanding, sampled from five existing benchmarks (Video-MME, LongVideoBench, LVBench, VideoMMMU, VSI-Bench) over 791 videos (all <= 60 min). It keeps only questions that need the video: every question that at least two of three models (Qwen2.5-VL-7B-Instruct, Qwen3-VL-8B-Instruct, InternVL3.5-8B) answer correctly from black frames was removed before sampling. Items were then selected to preserve the model ranking of the full benchmarks at ~5x lower cost.
Companion dataset with the video-dependent remainder of each source benchmark: soumyasj/video-benchmarks-nonblind.
data/meta_v-00000-of-00001.parquet - the 1,000 items (config meta_v, split test).videos/<source>/... - the 791 videos, referenced by the video column.| column | description |
|---|---|
id | <source_task>::<source_id> |
source | Video-MME, LongVideoBench, LVBench, VideoMMMU or VSI-Bench |
source_task | lmms-eval task name of the source (video_mmmu_perception etc.) |
source_id | native question id in the source benchmark |
video | repo-relative path of the video |
duration_s, duration_bucket | video length in seconds; short < 5 min, medium 5-20 min, long 20-60 min |
question, options, answer | options is empty for the 63 numerical VSI-Bench items; answer is the option letter or the number |
answer_type | multiple_choice (937) or numerical (63) |
categories | JSON string with the source-specific tags (task type, domain, duration group, discipline, ...; LVBench question_type is a list) |
blind_correct_models | how many of the three filter models solved the item from black frames (0 or 1) |
| source | items |
|---|---|
| Video-MME | 312 |
| LongVideoBench (val, video) | 240 |
| VideoMMMU (perception / comprehension / adaptation) | 60 / 60 / 60 |
| LVBench | 146 |
| VSI-Bench | 122 |
The task metav of lmms_eval_video evaluates this dataset directly
(one subtask per source, each scored with the source benchmark's own parser and metric); the first run downloads
this repo to $HF_HOME/meta_video:
python -m lmms_eval --model qwen2_5_vl --model_args pretrained=Qwen/Qwen2.5-VL-7B-Instruct,max_num_frames=32 \
--tasks metav --batch_size 1 --log_samples --output_path logs/metav
python eval_scripts/score_metav.py --write logs/metav/*/*_results.json
Accuracy per source (exact option match; VSI-Bench numerical items use mean relative accuracy), then the macro average over the five sources (VideoMMMU = mean of its three tracks).
The question lists are released under CC BY 4.0. The videos remain under the licenses and terms of their original benchmarks and sources (Video-MME, LongVideoBench, LVBench, VideoMMMU, VSI-Bench / ScanNet, ScanNet++, ARKitScenes); please follow those terms.
Meta-Video is a 1,000-question evaluation set for video understanding, sampled from five existing benchmarks (Video-MME, LongVideoBench, LVBench, VideoMMMU, VSI-Bench) over 791 videos (all <= 60 min). It keeps only questions that need the video: every question that at least two of three models (Qwen2.5-VL-7B-Instruct, Qwen3-VL-8B-Instruct, InternVL3.5-8B) answer correctly from black frames was removed before sampling. Items were then selected to preserve the model ranking of the full benchmarks at ~5x lower cost.
Companion dataset with the video-dependent remainder of each source benchmark: soumyasj/video-benchmarks-nonblind.
data/meta_v-00000-of-00001.parquet - the 1,000 items (config meta_v, split test).videos/<source>/... - the 791 videos, referenced by the video column.| column | description |
|---|---|
id | <source_task>::<source_id> |
source | Video-MME, LongVideoBench, LVBench, VideoMMMU or VSI-Bench |
source_task | lmms-eval task name of the source (video_mmmu_perception etc.) |
source_id | native question id in the source benchmark |
video | repo-relative path of the video |
duration_s, duration_bucket | video length in seconds; short < 5 min, medium 5-20 min, long 20-60 min |
question, options, answer | options is empty for the 63 numerical VSI-Bench items; answer is the option letter or the number |
answer_type | multiple_choice (937) or numerical (63) |
categories | JSON string with the source-specific tags (task type, domain, duration group, discipline, ...; LVBench question_type is a list) |
blind_correct_models | how many of the three filter models solved the item from black frames (0 or 1) |
| source | items |
|---|---|
| Video-MME | 312 |
| LongVideoBench (val, video) | 240 |
| VideoMMMU (perception / comprehension / adaptation) | 60 / 60 / 60 |
| LVBench | 146 |
| VSI-Bench | 122 |
The task metav of lmms_eval_video evaluates this dataset directly
(one subtask per source, each scored with the source benchmark's own parser and metric); the first run downloads
this repo to $HF_HOME/meta_video:
python -m lmms_eval --model qwen2_5_vl --model_args pretrained=Qwen/Qwen2.5-VL-7B-Instruct,max_num_frames=32 \
--tasks metav --batch_size 1 --log_samples --output_path logs/metav
python eval_scripts/score_metav.py --write logs/metav/*/*_results.json
Accuracy per source (exact option match; VSI-Bench numerical items use mean relative accuracy), then the macro average over the five sources (VideoMMMU = mean of its three tracks).
The question lists are released under CC BY 4.0. The videos remain under the licenses and terms of their original benchmarks and sources (Video-MME, LongVideoBench, LVBench, VideoMMMU, VSI-Bench / ScanNet, ScanNet++, ARKitScenes); please follow those terms.