JavisGPT: A Unified Multi-modal LLM for Sounding-Video Comprehension and Generation

TL;DR

We introduce JavisGPT, a multimodal LLM that can understand audiovisual inputs and simultaneously generate synchronized sounding videos in a unified model. We also curate the JavisInst-Omni dataset to facilitate instruction-tuning for comprehension and generation on sounding videos.

📰 News

• [2025.12.30] 🚀 We release the training dataset of JavisInst-Omni to support multimodal instruction tuning on sounding video comprehension and generation tasks, as well as MM-PreTrain and AV-FineTune datasets to enable preliminary multimodal alignment for LLMs.
• [2025.12.26] 🔥 We release the code of JavisGPT, with the preview JavisGPT-v0.1-7B-Instruct checkpoint at huggingface. Feel free to play with it!

The JavisInst-Omni Dataset

Introduction

JavisInst-Omni is constructed to enable flexible and interleaved audio-video comprehension and generation, including: (1) audio comprehension, (2) image/video comprehension, (3) joint audio-video comprehension, and (4) joint audio-video generation. The data sources come from various publicly available datasets, and we leverage GPT-4o to curate question-answer pairs to support multimodal instruction tuning. For more details please refer to our paper.

Usage

First, download the dataset from huggingface:

huggingface-cli download --repo-type dataset JavisVerse/JavisInst-Omni --local-dir /path/to/JavisInst-Omni

Then, unzip the .zip files in each subset of the entire JavisInst-Omni dataset:

cd /path/to/JavisInst-Omni
# the raw `.zip` files will be deleted with the `--purge` flag
python unzip.py --purge

Data source and QA pairs are organized with the stage3_mm_it.json meta file.

In addition, we cannot release the source data of TAVGBench due to policy issues. Instead, the video_ids (formatted with {youtube_id}_{start_time}_{end_time}) are provided in TAVGBench/video_ids.txt, and users can refer to TAVGBench to download raw videos.

Citation

If you find JavisGPT is useful and use it in your project, please kindly cite:

@inproceedings{liu2025javisgpt,
    title={JavisGPT: A Unified Multi-modal LLM for Sounding-Video Comprehension and Generation},
    author={Kai Liu and Jungang Li and Yuchong Sun and Shengqiong Wu and jianzhang gao and Daoan Zhang and Wei Zhang and Sheng Jin and Sicheng Yu and Geng Zhan and Jiayi Ji and Fan Zhou and Liang Zheng and Shuicheng YAN and Hao Fei and Tat-Seng Chua},
    booktitle={The Thirty-ninth Annual Conference on Neural Information Processing Systems},
    year={2025},
}