VideoRefer Suite: Advancing Spatial-Temporal Object Understanding with Video LLM

If you like our project, please give us a star ⭐ on Github for the latest update.

VideoRefer-Bench is a comprehensive benchmark to evaluate the object-level video understanding capabilities of a model, which consists of two sub-benchmarks: VideoRefer-Bench-D and VideoRefer-Bench-Q.

VideoRefer-Bench-D

The benchmark is designed to evaluate the description generation performance of video-based referring models. The benchmark comprises a total of 400 curated data entries. We curated the test set based on Panda-70M, employing the automatic pipeline, followed by a meticulous human check.

This benchmark covers four key aspects:

1. Subject Correspondence (SC): This dimension evaluates whether the subject of the generated description accurately corresponds to that specified in the ground truth.
2. Appearance Description (AD): This criterion assesses the accuracy of appearance-related details, including color, shape, texture, and other relevant visual attributes.
3. Temporal Description (TD): This aspect analyzes whether the representation of the object’s motion is consistent with the actual movements.
4. Hallucination Detection (HD): This facet identifies discrepancies by determining if the generated description includes any facts, actions, or elements absent from reality, like imaginative interpretations or incorrect inferences.

Data Format

For each object, we uniformly sampled 32 frames to generate the corresponding mask.

The data format organized in the benchmark json file is as below:

[
    {
        "id": 0,
        "video": "rLlzmcp3J6s_0:01:09.633_0:01:14.333.mp4",
        "caption": "The cub is a smaller, light colored lion. It is lying down and resting its head against the other lion. The cub looks calm and relaxed. It is the lion on the far left side of the frame.",
        "frame_idx": "36",
        "annotation":[
            {
                "2":{
                    "segmentation": {
                    }
                },
                "6":{
                    "segmentation": {
                    }
                },
                ...
            }
        ]
    }
]

• frame_idx: When using single-frame mask mode, we only use the single mask with the frame_idx.
• All the segmentations are in RLE format.

VideoRefer-Bench-Q

The benchmark is designed to evaluate the proficiency of MLLMs in interpreting video objects, including 1,000 high-quality multiple-choice questions. The benchmark covers five types of questions:

1. Basic Questions
2. Sequential Questions
3. Relationship Questions
4. Reasoning Questions
5. Future Predictions | Type | GPT-4o | GPT-4o-mini | InternVL2-26B | Qwen2-VL-7B | VideoRefer | | ---------------------- | -------- | ----------- | ------------- | ----------- | ---------- | | Basic Questions | 62.3 | 57.6 | 58.5 | 62.0 | 75.4 | | Sequential Questions | 74.5 | 67.1 | 63.5 | 69.6 | 68.6 | | Relationship Questions | 66.0 | 56.5 | 53.4 | 54.9 | 59.3 | | Reasoning Questions | 88.0 | 85.9 | 88.0 | 87.3 | 89.4 | | Reasoning Questions | 73.7 | 75.4 | 78.9 | 74.6 | 78.1 | | Average | 71.3 | 65.8 | 65.0 | 66.0 | 71.9 |

Data Format

For each object, we uniformly sampled 32 frames to generate the corresponding mask. The data format organized in the benchmark json file is as below:

[
    {
        "id": 0,
        "video": "DAVIS/JPEGImages/480p/aerobatics",
        "Question": "What is <object3><region> not wearing?",
        "type": "Basic Questions",
        "options": [
            "(A) A helmet",
            "(B) A hat",
            "(C) Sunglasses",
            "(D) A watch"
        ],
        "Answer": "(A) A helmet",
        "frame_idx": "57",
        "annotation":[
            {
                "0":{
                    "segmentation": {
                    }
                },
                "3":{
                    "segmentation": {
                    }
                },
                ...
            }
        ]
    }
]

• frame_idx: When using single-frame mask mode, we only use the single mask with the frame_idx.
• All the segmentations are in RLE format.