Video Detailed Caption Benchmark

Resources

• Website
• arXiv: Paper
• GitHub: Code
• Huggingface: AuroraCap Model
• Huggingface: VDC Benchmark
• Huggingface: Trainset

Features

Benchmark Collection and Processing

We building VDC upon Panda-70M, Ego4D, Mixkit, Pixabay, and Pexels. Structured detailed captions construction pipeline. We develop a structured detailed captions construction pipeline to generate extra detailed descriptions from various perspectives, significantly extending the length and enhancing the richness compared to previous benchmarks. The structured detailed captions includes the following categories:

• Camera caption. Describe the camera work in detail, including shot types, angles, movements, transitions, and any special effects used to enhance the video.
• Short caption. Summarize the video in one detailed sentence, capturing key actions and the overall mood.
• Background caption. Provide a detailed description of the background, including objects, location, weather, time, and any dynamic elements.
• Main Object caption. Give a thorough description of the main subject's actions, attributes, interactions, and movements throughout the video frames.
• Detailed caption. Generate a detailed, vivid caption for the video, covering all categories, ensuring it's engaging, informative, and rich enough for AI to recreate the video content.

Benchmarking video detailed captioning

AuroraCap achieves superior performance in video detailed captioning while utilizing significantly fewer visual tokens than other models, fully highlighting the efficiency of AuroraCap.

Quick Start

See Eval Docs.

Citation

@article{chai2024auroracap,
  title={AuroraCap: Efficient, Performant Video Detailed Captioning and a New Benchmark },
  author={Wenhao Chai, Enxin Song, Yilun Du, Chenlin Meng, Vashisht Madhavan, Omer Bar-Tal, Jeng-Neng Hwang, Saining Xie, Christopher D. Manning},
  journal={arXiv preprint arXiv:2410.03051},
  year={2024}
}