VANTAGE-BENCH

Video ANalysis Tasks Across Generalized Environments

Dataset Description

VANTAGE-BENCH is the first public benchmark purpose-built for evaluating visual understanding on video captured by fixed infrastructure cameras. It spans three real-world domains — warehouse, smart city / Intelligent Transportation Systems (ITS), and smart spaces — across six spatio-temporal video understanding tasks including video question answering (VQA), temporal grounding, dense video captioning, event verification, spatial grounding, and spatio-temporal tracking.

This dataset is for evaluation purposes only.

Dataset Owner(s)

NVIDIA Corporation

Dataset Creation Date

April 24, 2026

License/Terms of Use

This dataset is released under the NVIDIA Evaluation Data License.

Dataset Characterization

Data Collection Method
Hybrid: Human, Synthetic, Automated. Video data is sourced from vendor-provided footage (GoPro captures of warehouse and smart space environments), synthetic generation (DriveSim collision and multi-camera scenarios), and publicly scraped sources (Dubuque highway/ITS footage).

Labeling Method
Hybrid: Human, Synthetic, Pseudolabeled. Annotations for VQA, dense video captions, and temporal localization are primarily human-authored. Spatial grounding labels (2D/3D bounding boxes, referring expressions) use a combination of human annotation and pseudolabeling pipelines (detection + SAM for spatial pointing). Event verification labels are human-curated. Annotations are held server-side for evaluation only.

Directory Structure

VANTAGE-BENCH/
├── vqa/                     # Video question answering
├── dense_captioning/        # Dense video captioning
├── temporal_localization/   # Temporal localization
├── event_verification/      # Event verification
├── 2dbbox/                  # 2D object localization
├── referring/               # 2D referring expressions
├── pointing/                # 2D spatial pointing
├── tracking/                # Spatio-temporal tracking
└── README.md                # Dataset documentation and submission instructions

Evaluation

Tasks and Submission Formats

Submit predictions via the VANTAGE-Bench submission portal and track results on the official leaderboard.

Metric Notes

• Accuracy: Percentage of correct predictions.
• SODA-c: Metric for dense video captioning quality across event coverage and language quality.
• Macro F1: Unweighted mean of per-class F1 scores.
• F1@0.5: F1 score at an IoU threshold of 0.5.
• mIoU: Mean Intersection over Union — average overlap between predicted and ground-truth regions (spatial bounding boxes or temporal segments, depending on the task).
• AUC: Area under the ROC curve, measuring the model's ability to distinguish correct detections or tracks from incorrect ones across varying confidence thresholds.

Evaluation Server

The VANTAGE-Bench GitHub repository provides a sample evaluation pipeline for generating model predictions. Predictions are submitted through the submission portal and scored on the official leaderboard.

Dataset Format

Video (mp4) and Images (jpg).

Dataset Quantification

Total Entries (Annotations): 35,027 Total Media Samples (across tasks, with overlaps): 3,346 Total Data Storage: 42 GB

Potential Known Risks

• Ground truth annotations are not publicly released. All evaluation is performed server-side.
• Some warehouse videos are concatenated clips from longer recording sessions.

Citations

@inproceedings{Fujita2020SODA,
  author    = {Soichiro Fujita and Tsutomu Hirao and Hidetaka Kamigaito and Manabu Okumura and Masaaki Nagata},
  title     = {{SODA}: Story Oriented Dense Video Captioning Evaluation Framework},
  booktitle = {Proc. ECCV},
  year      = {2020}
}

@inproceedings{Fu2024BLINK,
  author    = {Xingyu Fu and Yushi Hu and Bangzheng Li and Yu Feng and Haoyu Wang and Xudong Lin and Dan Roth and Noah A. Smith and Wei-Chiu Ma and Ranjay Krishna},
  title     = {{BLINK}: Multimodal Large Language Models Can See but Not Perceive},
  booktitle = {Proc. ECCV},
  year      = {2024}
}

@article{Sun2025RefDrone,
  author    = {Zhichao Sun and Yuda Zou and Xian Sun and Yingchao Feng and Wenhui Diao and Menglong Yan and Kun Fu},
  title     = {{RefDrone}: A Challenging Benchmark for Referring Expression Comprehension in Drone Scenes},
  journal   = {arXiv preprint arXiv:2502.00392},
  year      = {2025}
}

References

• HuggingFace dataset: nvidia/PhysicalAI-VANTAGE-Bench
• Project website: vantage-bench.org

Ethical Considerations

NVIDIA believes Trustworthy AI is a shared responsibility and we have established policies and practices to enable development for a wide array of AI applications. When downloaded or used in accordance with our terms of service, developers should work with their internal developer teams to ensure this dataset meets requirements for the relevant industry and use case and addresses unforeseen product misuse. Please report quality, risk, security vulnerabilities or NVIDIA AI Concerns here.

Changelog

See CHANGELOG.md for release history.