Dataset Card for T2AV-Compass

Dataset Details

Dataset Description

T2AV-Compass is a unified benchmark for evaluating Text-to-Audio-Video (T2AV) generation, targeting not only unimodal quality (video/audio) but also cross-modal alignment & synchronization, complex instruction following, and perceptual realism grounded in physical/common-sense constraints.

Text-to-Audio-Video (T2AV) generation aims to synthesize temporally coherent video and semantically synchronized audio from natural language, yet its evaluation remains fragmented, often relying on unimodal metrics or narrowly scoped benchmarks that fail to capture cross-modal alignment, instruction following, and perceptual realism under complex prompts.

This dataset includes 500 taxonomy-driven prompts and fine-grained checklists for an MLLM-as-a-Judge protocol.

• Curated by: NJU-LINK Team (Nanjing University), Kling Team (Kuaishou Technology), Institute of Automation (Chinese Academy of Sciences)
• Language(s) (NLP): English
• License: CC BY-NC-SA 4.0

Dataset Sources

• Repository: github.com/NJU-LINK/T2AV-Compass
• Project Page: nju-link.github.io/T2AV-Compass
• Paper:

Uses

Direct Use

T2AV-Compass is designed for:

• Benchmarking T2AV models: Evaluate text-to-audio-video generation systems across multiple dimensions
• Research: Study cross-modal alignment, instruction following, and perceptual realism in generative models
• Model development: Identify failure modes and guide improvements in T2AV systems

Out-of-Scope Use

• This dataset is intended for research and evaluation purposes only
• Not intended for training generative models directly
• Should not be used for commercial applications without proper licensing

Dataset Structure

Files

• data/prompts_with_checklist.json: Core benchmark data (500 prompts + checklists)

Data Schema

Each sample is a JSON object with the following fields:

Model Adaptation

• End-to-end T2AV models (e.g., Veo, Kling): Use prompt
• Two-stage / modular pipelines:
  • Video model: video_prompt
  • Audio model: audio_prompt
  • TTS / speech: speech_prompt

Dataset Creation

Curation Rationale

Existing T2AV evaluation benchmarks often necessitate trade-offs between:

1. Fine-grained semantic taxonomy
2. Scalable, interpretable judging signals
3. Balanced coverage of diverse coupling phenomena (e.g., multi-source sound mixing, physical plausibility, and commonsense consistency)

T2AV-Compass addresses these limitations with a taxonomy-driven curation pipeline and dual-level evaluation framework.

Source Data

Data Collection and Processing

The dataset is constructed through a three-stage pipeline:

1. Data Collection: Aggregated raw prompts from high-quality sources including VidProM, Kling AI community, LMArena, and Shot2Story. Semantic clustering with cosine similarity threshold of 0.8 was applied for deduplication.

2. Prompt Refinement and Alignment: Used Gemini-2.5-Pro to restructure and enrich prompts with visual subjects, motion dynamics, acoustic events, and cinematographic constraints. Manual audit filtered out static scenes or illogical compositions, resulting in 400 complex prompts.

3. Real-world Video Inversion: Selected 100 diverse, high-fidelity video clips (4–10s) from YouTube with dense captioning via Gemini-2.5-Pro and human-in-the-loop verification.

Who are the source data producers?

• VidProM dataset
• Kling AI community
• LMArena
• Shot2Story dataset
• YouTube (for video inversion)

Annotations

Annotation process

Each prompt is annotated with:

• Taxonomy labels (subject, scenario, sound type, camera movement)
• Structured checklists for MLLM-as-a-Judge evaluation
• Separate video/audio/speech prompts for modular systems

Who are the annotators?

Research team members with expertise in multimodal generation and evaluation.

Evaluation Framework

Objective Metrics

Subjective Metrics (MLLM-as-a-Judge)

Instruction Following (IF) - 7 dimensions, 17 sub-dimensions:

• Attribute: Look, Quantity
• Dynamics: Motion, Interaction, Transformation, Camera Motion
• Cinematography: Light, Frame, Color Grading
• Aesthetics: Style, Mood
• Relations: Spatial, Logical
• World Knowledge: Factual Knowledge
• Sound: Sound Effects, Speech, Music

Realism - 5 metrics:

• Video: MSS (Motion Smoothness), OIS (Object Integrity), TCS (Temporal Coherence)
• Audio: AAS (Acoustic Artifacts), MTC (Material-Timbre Consistency)

Bias, Risks, and Limitations

• The dataset focuses on English prompts only
• Prompts are designed for complex, challenging scenarios which may not represent typical user queries
• Evaluation metrics require specific pretrained models (DOVER++, CLAP, ImageBind, etc.)

Recommendations

Users should:

• Use appropriate evaluation models as specified in the paper
• Consider the benchmark's focus on challenging scenarios when interpreting results
• Follow the MLLM-as-a-Judge protocol for subjective evaluation

Citation

If you find this work useful, please cite:

@misc{cao2025t2avcompass,
  title         = {T2AV-Compass: Towards Unified Evaluation for Text-to-Audio-Video Generation},
  author        = {Cao, Zhe and Wang, Tao and Wang, Jiaming and Wang, Yanghai and Zhang, Yuanxing and Chen, Jialu and Deng, Miao and Wang, Jiahao and Guo, Yubin and Liao, Chenxi and Zhang, Yize and Zhang, Zhaoxiang and Liu, Jiaheng},
  year          = {2025},
  eprint        = {2512.21094},
  archivePrefix = {arXiv},
  primaryClass  = {cs.CV},
  url           = {https://arxiv.org/abs/2512.21094},
}

Dataset Card Authors

NJU-LINK Team, Nanjing University

Dataset Card Contact

• zhecao@smail.nju.edu.cn
• liujiaheng@nju.edu.cn