AVE-Compass

Project Page | Paper | Evaluation Code

AVE-Compass is a benchmark for evaluating instruction-based audio-video editing abilities. It targets realistic editing requests where visual and audio streams are tightly coupled, such as changing a visible event together with its sound, preserving background ambience during visual edits, or editing speech while maintaining audio-visual consistency.

This dataset release contains the source videos, edit instruction JSON files, and checklist JSON files used by AVE-Compass.

Benchmark Overview

AVE-Compass contains:

• 145 curated source videos.
• 196 human-verified audio-video editing instructions.
• 2,688 fine-grained checklist items for diagnostic evaluation.
• 28 fine-grained editing operation types.

The benchmark covers four major editing branches:

• Joint audio-visual editing: edits that require coordinated changes across video and audio.
• Speech editing: edits involving spoken content, speaker identity, or speech-related audio-video consistency.
• Video-only editing: visual edits where the audio stream should be preserved.
• Audio-only editing: audio edits where the visual stream should be preserved.

What AVE-Compass Evaluates

AVE-Compass evaluates edited audio-video outputs along complementary dimensions:

• Editing Intent: the primary metric, measuring whether the output both follows the instruction and preserves non-target content.
• Instruction Following: whether the requested edit is correctly executed.
• Fidelity Preserving: whether non-target visual and audio content remains faithful to the source.
• Realism: whether the edited audio-video result is natural, coherent, and free of obvious artifacts.

The paper also reports automated metrics for cross-modal, visual, and audio quality, including AV Sync, Lip Sync, Video Aesthetic, Subject Consistency, Motion Smoothness, Audio Aesthetic, and Speech Quality.

Leaderboard

Models are ranked by Overall Editing Intent, the primary metric of AVE-Compass. Scores are reported on a 0-100 scale, and higher is better.

*Gemini-Omni misses 16 speech edits due to content moderation.

Dataset Files

AVE-Compass/
  assets/
    bench.png              # benchmark overview image
    evaluation_matrix.png  # evaluation matrix rendered from the paper
  videos/
    *.mp4              # 145 source videos
    metadata.jsonl     # 196 Dataset Viewer rows
  edit_instructions/  # 196 edit instruction JSON files
  checklists/         # 196 checklist JSON files

The Dataset Viewer uses videos/metadata.jsonl to display each source video together with its edit instruction and instruction_index.

Edit Instruction JSON Format

Each file in edit_instructions/ corresponds to one edit instruction:

{
  "video": "example.mp4",
  "task": "example",
  "instruction_index": 1,
  "total_instructions_for_video": 1,
  "instruction": {
    "category_label": "joint",
    "category": "J1",
    "operation": "J1.1 New Source Insertion",
    "prompt_en": "Add ...",
    "audio_label": {
      "audio_op": "add",
      "sound_type": "event_sfx",
      "edit_aspect": "content"
    },
    "difficulty": {
      "d1_object_localization": "hard",
      "d1_reason": "...",
      "d2_audio_complexity": "complex",
      "d3_cross_modal_linkage": "explicit",
      "d3_reason": "..."
    }
  }
}

Important fields:

• video: source video filename.
• instruction_index: 1-based index of the edit instruction for the source video.
• prompt_en: English edit instruction.
• category_label: one of joint, speech, video_only, or audio_only.
• audio_label: structured annotation of the audio-side edit target.
• difficulty: difficulty annotations for object localization, audio complexity, and cross-modal linkage.

Checklist JSON Format

Each file in checklists/ corresponds to the edit instruction with the same video and instruction_index. Checklist files contain atomic Yes/No questions for evaluating instruction following and fidelity preservation.

Typical fields include:

• video: source video filename.
• instruction_index: 1-based edit instruction index for the source video.
• edit_prompt: the edit instruction.
• edit_category: editing branch.
• questions: modality-tagged diagnostic questions.

Each checklist question includes:

• question_id
• dimension
• subdimension
• modality_tag
• question

Data Construction

AVE-Compass was constructed through a human-in-the-loop pipeline. Candidate edit instructions were generated from structured source-video descriptions using modality-aware LLM generators. A critic model filtered out unnatural, infeasible, or ambiguous instructions, and human annotators verified naturalness, executability, and target specificity. The verified instructions were then converted into fine-grained checklist items, followed by human deduplication and refinement.

Usage Notes

• Pair each edit instruction with the source video indicated by its video field.
• Pair each checklist with the edit instruction sharing the same video and instruction_index.

Citation

If you use AVE-Compass, please cite:

@article{wen2026avecompass,
  title = {AVE-Compass: Towards Holistic Evaluation for Audio-Video Editing Abilities},
  author = {Wen, Yuqing and Huang, Yukai and Xie, Qianqian and Wu, Jiangtao and Lin, Yibin and Gu, Yikai and Chen, Jialu and Zhang, Yuanxing and Liu, Jiaheng},
  year = {2026},
  eprint = {2607.24821},
  archivePrefix = {arXiv},
  primaryClass = {cs.MM},
  url = {https://arxiv.org/abs/2607.24821}
}