VET-Bench: Visual Entity Tracking Benchmark

Paper | Project Page | GitHub

VET-Bench is a synthetic diagnostic benchmark simulating the realistic shell game with visually indistinguishable objects that forces models to track entities through spatiotemporal continuity. The task is easy for human but difficult for current VLMs. State-of-the-art VLMs perform at random chance, while our proposed Molmo2-SGCoT achieves over 90% accuracy.

Dataset Overview

Cup Game

A ball is placed under one of three colored cups. The cups are shuffled 5 times, and the model must identify which cup (Left / Middle / Right) contains the ball at the end.

Card Game

Three playing cards are shown face-up, then turned face-down and shuffled 5 times. The model must determine the final position (Left / Middle / Right) of a specific card (the Queen of Hearts).

Data Structure

vetbench/
├── cup/
│   ├── cup_001.mp4 ... cup_050.mp4   # 50 cup game videos
│   ├── cup.json                       # Full metadata (settings, swap sequences, ground truth)
│   └── QA.json                        # Evaluation-ready QA pairs
├── card/
│   ├── card_001.mp4 ... card_050.mp4  # 50 card game videos
│   ├── card.json                      # Full metadata (settings, swap sequences, ground truth)
│   └── QA.json                        # Evaluation-ready QA pairs
└── README.md

Metadata Format (cup.json / card.json)

Each entry contains:

• video: filename
• fps, total_frames, resolution: video properties
• task: "cup" or "card"
• game_settings: number of objects, swap count, speed
• ground_truth: correct final position(s)
• initial: starting arrangement
• intermediate: arrangement after each swap

QA Format (QA.json)

Each entry contains:

• video, question, answer (A/B/C)
• Questions follow the format:

Cup: "Which cup contains the ball at the end of the video? (A) Left (B) Middle (C) Right"

Card: "Where is the Queen of Hearts at the end of the video? (A) Left (B) Middle (C) Right"

Benchmark Results

Random-chance baseline for 3-way multiple choice: 33.3%.

Most models perform near random chance, demonstrating that visual object tracking remains a significant challenge for current Video-LLMs. The exception is Molmo2-SGCoT, which uses a Spatially-Grounded Chain-of-Thought prompting strategy to explicitly track object coordinates frame by frame.

Usage

Loading with Hugging Face

from huggingface_hub import snapshot_download

snapshot_download(
    repo_id="tiedong/vetbench",
    repo_type="dataset",
    local_dir="vetbench"
)

License

This dataset is released under the MIT License.