AV-DPO

AV-DPO is the audio-video preference dataset used in Stage 3 of JavisDiT++. It contains chosen-rejected sounding-video pairs selected jointly across audio quality and text alignment, video quality and text alignment, and audio-video semantic and temporal alignment.

Release summary

Each MP4 contains both H.264 video and an AAC audio track. Separate audio files are unnecessary for JavisDiT++ training and are not included.

Construction

We curated a prompt pool separate from the AV-SFT training set. For every prompt, the Stage 2 reference model generated three audio-video candidates and the corresponding ground-truth sounding video was added as a fourth candidate. Candidates were evaluated with the following modality-aware rewards:

• Audio: AudioBox audio quality and ImageBind text-audio alignment.
• Video: VideoAlign visual/motion quality and ImageBind text-video alignment.
• Audio-video: ImageBind semantic alignment and Synchformer temporal synchrony.

Each metric was normalized, metrics were averaged within their modality, and a pair was retained only when the chosen candidate outranked the rejected candidate in all three modality groups. This produces 23,671 available pairs, reported as approximately 25K in the paper.

Files

AV-DPO/
├── README.md
├── SHA256SUMS
├── manifest.json
├── train.csv
├── train_generated_only.csv
├── prompt_pool_video_ids.txt
├── required_ground_truth_video_ids.txt
└── data_zips/
    ├── generated_240p.zip
    ├── generated_360p.zip
    └── generated_480p.zip

train.csv is the full paper setting. Because the source TAVGBench/YouTube videos cannot be redistributed, its ground-truth entries point to data/AV-DPO/data/tavgbbench/<video_id>.mp4, which users must resolve locally. train_generated_only.csv contains only pairs whose chosen and rejected media are both included in this repository, and therefore works immediately after download. prompt_pool_video_ids.txt records the locally available prompt pool; required_ground_truth_video_ids.txt is the smaller set needed by the final full preference split.

Schema

Download and training

cd /path/to/JavisDiT
hf download --repo-type dataset JavisVerse/AV-DPO \
  --local-dir data/AV-DPO

(cd data/AV-DPO && sha256sum -c SHA256SUMS)  # macOS: shasum -a 256 -c
for archive in data/AV-DPO/data_zips/*.zip; do unzip -q "$archive" -d data/AV-DPO; done

To train immediately on the self-contained subset:

torchrun --standalone --nproc_per_node 8 \
  scripts/train.py \
  configs/javisdit-v1-0/train/stage3_audio_video_dpo.py \
  --data-path data/AV-DPO/train_generated_only.csv

To reproduce the full paper setting, first obtain the source videos following the TAVGBench instructions, then make them available at the metadata's expected local path without copying them:

mkdir -p data/AV-DPO/data
ln -s /path/to/TAVGBench/videos data/AV-DPO/data/tavgbbench

torchrun --standalone --nproc_per_node 8 \
  scripts/train.py \
  configs/javisdit-v1-0/train/stage3_audio_video_dpo.py \
  --data-path data/AV-DPO/train.csv

The metadata follows JavisDiT's native path/path_reject DPO schema. Its paths assume the dataset is downloaded to JavisDiT/data/AV-DPO and training is launched from the JavisDiT repository root.

The Stage 3 configuration sets use_audio_in_video=True, so it reads audio directly from each MP4 and does not require extraction. If standalone 16 kHz WAV files are needed for inspection or another tool, use JavisDiT's dataset utilities to extract every released generated track in batch:

python -m tools.datasets.convert video data/AV-DPO/data/generated \
  --output data/AV-DPO/generated_videos.csv
python -m tools.datasets.datautil data/AV-DPO/generated_videos.csv \
  --extract-audio --audio-sr 16000

This writes each WAV next to its source MP4 and records the paths in data/AV-DPO/generated_videos_au_sr16000.csv. It is an optional local step; WAV files are not part of the Hugging Face release.

Intended use and limitations

AV-DPO is intended for research on preference alignment in joint audio-video generation. Rankings are produced by automatic reward models rather than direct human pair annotation and may inherit their biases or errors. Generated media may contain visual or acoustic artifacts. Users should review samples and apply appropriate safety filtering before downstream deployment.

Ground-truth TAVGBench/YouTube media are not included. This repository only publishes their clip identifiers and preference metadata; users are responsible for complying with upstream licenses, platform terms, and applicable law when obtaining source media.

The release uses separate terms for different components:

• Original preference metadata and ID lists: CC BY 4.0.
• Model-generated sounding videos: CC BY 4.0.
• TAVGBench/YouTube source media: not distributed and not covered by this repository's licenses.

Citation

@inproceedings{liu2026javisdit++,
  title     = {JavisDiT++: Unified Modeling and Optimization for Joint Audio-Video Generation},
  author    = {Liu, Kai and Zheng, Yanhao and Wang, Kai and Wu, Shengqiong and Zhang, Rongjunchen and Luo, Jiebo and Hatzinakos, Dimitrios and Liu, Ziwei and Fei, Hao and Chua, Tat-Seng},
  booktitle = {The Fourteenth International Conference on Learning Representations},
  year      = {2026},
}