[ACL'25 Oral] UrbanVideo-Bench: Benchmarking Vision-Language Models on Embodied Intelligence with Video Data in Urban Spaces

This repository contains the dataset introduced in the paper, consisting of two parts: 5k+ multiple-choice question-answering (MCQ) data and 1k+ video clips.

• Arxiv: https://arxiv.org/pdf/2503.06157
• Project: https://embodiedcity.github.io/UrbanVideo-Bench/
• Code: https://github.com/EmbodiedCity/UrbanVideo-Bench.code

Dataset Description

The benchmark is designed to evaluate whether video-large language models (Video-LLMs) can naturally process continuous first-person visual observations like humans, enabling recall, perception, reasoning, and navigation.

• Text Data: Available in the file MCQ.parquet.
• Video Data: Stored in the videos folder.

The MCQ.parquet contains the following fields:

For example, we evaluate the performance of 17 popular Video-LLMs on the dataset. For detailed results and analysis, please refer to the paper.

Citation

@inproceedings{zhao-etal-2025-urbanvideo,
    title = "{U}rban{V}ideo-Bench: Benchmarking Vision-Language Models on Embodied Intelligence with Video Data in Urban Spaces",
    author = "Zhao, Baining  and
      Fang, Jianjie  and
      Dai, Zichao  and
      Wang, Ziyou  and
      Zha, Jirong  and
      Zhang, Weichen  and
      Gao, Chen  and
      Wang, Yue  and
      Cui, Jinqiang  and
      Chen, Xinlei  and
      Li, Yong",
    editor = "Che, Wanxiang  and
      Nabende, Joyce  and
      Shutova, Ekaterina  and
      Pilehvar, Mohammad Taher",
    booktitle = "Proceedings of the 63rd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers)",
    month = jul,
    year = "2025",
    address = "Vienna, Austria",
    publisher = "Association for Computational Linguistics",
    url = "https://aclanthology.org/2025.acl-long.1558/",
    doi = "10.18653/v1/2025.acl-long.1558",
    pages = "32400--32423",
    ISBN = "979-8-89176-251-0"
}