Yinan Chen ^1★ · Chuming Lin ^2★ · Zhennan Chen ³ · Yuxiang Zeng ⁴ · Junwei Zhu ² ·
Yali Bi ¹ · Xijie Huang ⁵ · Chengming Xu ² · Donghao Luo ² · Zhucun Xue ¹ ·
Xiaobin Hu ⁶ · Chengjie Wang ² · Yong Liu ¹ · Jiangning Zhang ^1,2📧 · Shuicheng Yan ⁶

¹Zhejiang University     ²YouTu Lab, Tencent     ³Nanjing University
⁴University of Auckland     ⁵Fudan University     ⁶National University of Singapore

:blush: Continuous Updates

This repository is the official implementation of JAVEdit: Joint Audio-Visual Instruction-Guided Video Editing with Agentic Data Curation. It collects the dataset, model, and benchmark resources for instruction-guided joint audio-visual video editing. If you find any work missing or have any suggestions, feel free to open a pull request or contact us.

✨ Highlight!!!

While instruction-based video editing has made significant progress, joint audio-visual editing remains constrained by the absence of dedicated datasets and benchmarks. We bridge this gap with three tightly-coupled contributions:

1. JAVEdit-100k — the first large-scale joint audio-visual editing dataset: ~103K high-quality, human-centric editing triplets across five categories (Subject Editing, Background Editing, Subject Removal, Subject Addition, Speech Editing), all at 1280×720, 121 frames, 25 FPS, paired with free-form natural-language instructions.
2. Agent-in-the-loop quality control: a scalable, fully automated curation mechanism (Inspector + Orchestrator agents with a shared Problem Pattern Library) that detects failures, diagnoses root causes, and repairs the pipeline — raising the qualification rate from 36% to 83% without manual bottlenecks.
3. JAVEditBench — a human-aligned benchmark: 150 curated source videos with manually reviewed instructions, evaluated by six metrics across five dimensions that jointly assess visual–audio quality, instruction compliance, and video fidelity (Spearman's ρ ≥ 0.80 with human preference).
4. JAVEdit — a strong baseline: obtained by fine-tuning LTX-2.3 with LoRA on JAVEdit-100k, outperforming all baselines on five of six JAVEditBench metrics, with a 26% relative gain in audio-visual synchrony over the strongest sequential alternative.

:mailbox_with_mail: Summary of Contents

• Introduction
• Highlight
• Data Pipeline
• Benchmark Statistics
• Installation
  • Install requirements
  • Download pretrained checkpoints
  • Download JAVEdit-100k dataset
• Usage
  • Generate target videos
  • Run JAVEditBench evaluation
• Experiments
• Citation
• Contact

:movie_camera: Data Pipeline

Data construction pipeline of JAVEdit-100k. Source videos from OpenHumanVid, VIDGEN-1M, and VGGSound undergo four stages: (1) Preprocessing — basic quality filtering (SyncNet A/V-sync + Koala-36M VTSS aesthetics), dense captioning (Qwen3-Omni), and audio source separation (SAM-Audio) into disentangled voice / music / ambient streams; (2) Instruction Generation — task selection, balanced least-frequently-used topic sampling from a curated vocabulary bank, and paired visual + audio instruction generation (Qwen3-235B); (3) Editing Pipelines — four dedicated pipelines covering five categories; (4) Agent-in-the-loop Quality Control — closed-loop detect → diagnose → repair → re-check.

Agent-in-the-loop Quality Control

An Inspector agent (Gemini) examines sampled outputs and produces structured quality reports, while an Orchestrator agent (Claude) classifies failures into three levels — L1 Systemic, L2 Modular, and L3 Instance — and applies targeted fixes. Verified solutions are stored in a Problem Pattern Library for cross-pipeline reuse, raising the overall qualification rate from 36% to 83% over three rounds.

:sunflower: Benchmark Statistics

Statistical distributions of JAVEdit-100k and JAVEditBench. JAVEdit-100k is the only dataset that jointly covers audio and visual editing with free-form natural-language instructions:

:hammer: Installation

1. Install requirements

git clone git@github.com:RyanChenYN/JAVEdit.git
cd JAVEdit
conda create -n javeditbench python=3.12 -y
conda activate javeditbench
pip install -r requirements.txt

System dependencies:

• ffmpeg and ffprobe must be available on PATH.
• A CUDA driver compatible with CUDA 12.8 is required for the tested PyTorch/vLLM stack.
• The MLLM-based metrics were tested with vllm==0.11.1; newer vLLM releases may require CUDA 13 and are not recommended for this setup.

2. Download pretrained checkpoints

All checkpoint locations are configured in metrics/path.yml. Download the following pretrained models and update the corresponding paths:

• Qwen/Qwen3-Omni-30B-A3B-Thinking — instruction compliance / video fidelity / AV-quality judge
• Koala-36M/Training_Suitability_Assessment — VTSS visual quality
• sarulab-speech/UTMOSv2 — UTMOSv2 audio quality (fold0_s42_best_model.pth)
• ByteDance/LatentSync — SyncNet lip-sync weight (latentsync_syncnet.pt). Face detection and 106-point landmarks use the insightface buffalo_l pack (det_10g.onnx + 2d106det.onnx), downloaded automatically by insightface on first run into <model_root>/models/buffalo_l/.

After downloading, edit metrics/path.yml to point each entry to your local checkpoint directory. All paths are resolved relative to the location of path.yml.

3. Download the JAVEdit-100k dataset

The full training dataset and the JAVEditBench test set are hosted on Hugging Face:

huggingface-cli download --repo-type dataset --resume-download Coraxor/JAVEdit-100k --local-dir $YOUR_LOCAL_PATH

The benchmark metadata file benchmark_150.csv (150 rows, 147 unique videos) is already shipped with this repository at the project root, so you do not need to download it from Hugging Face. If you only need the JAVEditBench source videos for evaluation (≈437 MiB), pull the tarball alone instead of the full dataset and extract it next to the CSV (i.e., into the repo root):

# From the repo root
huggingface-cli download --repo-type dataset \
    Coraxor/JAVEdit-100k benchmark_videos.tar.gz \
    --local-dir .

# Extract — produces ./benchmark_videos/ with 147 .mp4 files alongside benchmark_150.csv
tar -xzf benchmark_videos.tar.gz

The CSV references each source video as a relative path benchmark_videos/<name>.mp4, so once the archive is extracted in the repo root the evaluation pipeline resolves paths automatically.

:muscle: Usage

1. Generate target videos

Run your own joint audio-visual editing model on the JAVEditBench source videos to produce edited (target) videos. Each edited video is matched to its source by the 32-character hash prefix in the filename, e.g.:

0a16d2122d7b9e9fe6c2a6aa65b658e6.mp4                          # source
0a16d2122d7b9e9fe6c2a6aa65b658e6_0_edited_with_audio.mp4      # target

The benchmark CSV provides at least the following columns:

video,task,prompt,detailed_prompt_cn

2. Run JAVEditBench evaluation

JAVEditBench provides a unified entry point for six metrics across five dimensions:

Run all metrics:

cd metrics

VLLM_WORKER_MULTIPROC_METHOD=spawn python evaluate.py \
    --video_dir /path/to/edited_videos \
    --bench_csv ../benchmark_150.csv \
    --output_path /path/to/eval_results \
    --metric syncnet vtss utmos av_quality instruction_compliance video_fidelity \
    --num_gpus 8 \
    --name javedit_eval

Run only the lightweight (non-MLLM) metrics on a single GPU:

python evaluate.py \
    --video_dir /path/to/edited_videos \
    --bench_csv ../benchmark_150.csv \
    --metric utmos \
    --num_gpus 1 \
    --name javedit_utmos

Each run writes a detailed *_results.json and a *_summary.csv to --output_path; runtime logs are written to metrics/logs/. Use VLLM_WORKER_MULTIPROC_METHOD=spawn whenever the Qwen3-Omni metrics are involved.

:bar_chart: Experiments

Quantitative Comparison

Quantitative comparison on JAVEditBench across five evaluation dimensions. JAVEdit ranks first on five of six metrics. Best results are bolded.

Compared with the strongest Sequential cascade (Kiwi-Edit + HunyuanVideo-Foley), joint modeling yields a 26% relative gain in audio-visual synchrony, validating the necessity of joint audio-visual modeling and agent-curated data.

Qualitative Comparison

Qualitative comparison on JAVEditBench. Rows correspond to the five editing categories; columns show the source video and the outputs of each method. JAVEdit consistently generates edits that are visually coherent, semantically faithful to the instruction, and temporally synchronized across both modalities.

:black_nib: Citation

If you find JAVEdit useful for your research, please consider giving a star ⭐ and citation 📝 :)

@article{chen2026javedit,
  title={JAVEdit: Joint Audio-Visual Instruction-Guided Video Editing with Agentic Data Curation},
  author={Chen, Yinan and Lin, Chuming and Chen, Zhennan and Zeng, Yuxiang and Zhu, Junwei and Bi, Yali and Huang, Xijie and Xu, Chengming and Luo, Donghao and Xue, Zhucun and Hu, Xiaobin and Wang, Chengjie and Liu, Yong and Zhang, Jiangning and Yan, Shuicheng},
  journal={arXiv preprint arXiv:2606.03168},
  year={2026}
}

✉️ Contact

yinan.chen@zju.edu.cn

:pray: Acknowledgements

JAVEdit is built upon and vendors components from many excellent open-source projects, including LTX-Video, Qwen3-Omni, Qwen3, Koala-36M, UTMOSv2, LatentSync, SAM-Audio, HunyuanImage-3.0, Wan2.2-Animate, OpenVE-3M, HunyuanVideo-Foley, MiniMax-Remover, and SAM3. We sincerely thank the authors for their contributions to the community.