Yinan Chen 1★ · Chuming Lin 2★ · Zhennan Chen 3 · Yuxiang Zeng 4 · Junwei Zhu 2 · Yali Bi 1 · Xijie Huang 5 · Chengming Xu 2 · Donghao Luo 2 · Zhucun Xue 1 · Xiaobin Hu 6 · Chengjie Wang 2 · Yong Liu 1 · Jiangning Zhang 1,2 📧 · Shuicheng Yan 6

1 Zhejiang University 2 YouTu Lab, Tencent 3 Nanjing University 4 University of Auckland 5 Fudan University 6 National University of Singapore

😊 Dataset Introduction

JAVEdit-100k is the official dataset accompanying 🎞️ JAVEdit: Joint Audio-Visual Instruction-Guided Video Editing with Agentic Data Curation.

While instruction-based video editing has progressed rapidly, joint audio–visual editing is still constrained by the absence of dedicated datasets and benchmarks. JAVEdit-100k is the first large-scale dataset for instruction-guided joint audio–visual editing, comprising approximately 103K human-centric editing triplets across five categories — Subject Editing, Background Editing, Subject Removal, Subject Addition, Speech Editing — all at 1280×720, 121 frames, 25 FPS, paired with free-form natural-language instructions.

This repository hosts both the full training set (~103K triplets, packed as multi-part archives) and JAVEditBench, a 150-sample human-aligned evaluation benchmark.

🗂️ Repository Layout

The repo is organized in two layers — a lightweight case preview for browsing, and the full archives for downloading the complete dataset.

1. Case folders (quick preview)

These four folders correspond to the four dedicated editing pipelines described in the paper. They are not the full data — each contains only a small number of triplets so you can quickly inspect what JAVEdit-100k looks like without downloading the full archives.

2. Full dataset archives

The benchmark metadata CSV (benchmark_150.csv) is shipped together with the JAVEdit GitHub repo and is not mirrored here.

🎥 Data Pipeline

Source videos from OpenHumanVid, VIDGEN-1M, and VGGSound are processed in four stages:

1. Preprocessing — quality filtering (SyncNet A/V-sync + Koala-36M VTSS aesthetics), dense captioning (Qwen3-Omni), and audio source separation (SAM-Audio) into disentangled voice / music / ambient streams.
2. Instruction Generation — task selection, balanced least-frequently-used topic sampling from a curated vocabulary bank, and paired visual + audio instruction generation (Qwen3-235B).
3. Editing Pipelines — four dedicated pipelines covering five categories (corresponding to the background/, person/, remove/, talk/ folders above).
4. Agent-in-the-loop Quality Control — an Inspector agent (Gemini) examines sampled outputs, an Orchestrator agent (Claude) classifies failures and applies targeted fixes, with verified solutions stored in a shared Problem Pattern Library. The closed loop raises the qualification rate from 36% to 83% over three rounds.

🌈 Comparison

JAVEdit-100k is the only dataset that jointly covers audio and visual editing with free-form natural-language instructions:

⚙️ Usage

Quick preview (cases only)

Browse the four case folders directly in the Hugging Face web UI, or fetch them locally:

huggingface-cli download --repo-type dataset Coraxor/JAVEdit-100k \
    --include "background/*" "person/*" "remove/*" "talk/*" \
    --local-dir $YOUR_LOCAL_PATH

JAVEditBench (evaluation, ~437 MiB)

git clone https://github.com/RyanChenYN/JAVEdit.git
cd JAVEdit

huggingface-cli download --repo-type dataset \
    Coraxor/JAVEdit-100k benchmark_videos.tar.gz \
    --local-dir .
tar -xzf benchmark_videos.tar.gz

After extraction, benchmark_videos/ sits next to benchmark_150.csv (already in the GitHub repo). The CSV uses relative paths benchmark_videos/<name>.mp4, so the evaluation pipeline resolves paths automatically — no extra configuration needed.

Full dataset (~103K triplets)

# 1. Download every archive in this repo
huggingface-cli download --repo-type dataset --resume-download \
    Coraxor/JAVEdit-100k --local-dir $YOUR_LOCAL_PATH
cd $YOUR_LOCAL_PATH

# 2. Reassemble the multi-part edited-videos archive and extract
cat edit_videos.tar.gz.* | tar -xzf -

# 3. Extract the rest
tar -xzf source_videos.tar.gz
tar -xzf edit_instruction.tar.gz

For slow networks, install hf_transfer and set HF_HUB_ENABLE_HF_TRANSFER=1 to accelerate downloading:

pip install hf_transfer
export HF_HUB_ENABLE_HF_TRANSFER=1

⚠️ Disclaimer

• Source videos in JAVEdit-100k are derived from publicly available datasets (OpenHumanVid, VIDGEN-1M, VGGSound). The dataset is released solely for academic research and educational purposes under the MIT license.
• All original videos belong to their respective copyright holders and hosting platforms. Users must comply with the licenses and usage terms of the upstream sources.
• Edited videos are generated by automated editing pipelines. Despite the agent-in-the-loop quality control, residual artifacts or imperfect outputs may still occur.

🖊️ Citation

If you find JAVEdit useful for your research, please consider giving us a like ❤️ and a citation 📝:

@article{chen2026javedit,
  title={JAVEdit: Joint Audio-Visual Instruction-Guided Video Editing with Agentic Data Curation},
  author={Chen, Yinan and Lin, Chuming and Chen, Zhennan and Zeng, Yuxiang and Zhu, Junwei and Bi, Yali and Huang, Xijie and Xu, Chengming and Luo, Donghao and Xue, Zhucun and Hu, Xiaobin and Wang, Chengjie and Liu, Yong and Zhang, Jiangning and Yan, Shuicheng},
  journal={arXiv preprint arXiv:2606.03168},
  year={2026}
}

---

✉️ Contact

yinan.chen@zju.edu.cn