JavisGPT: A Unified Multi-modal LLM for Sounding-Video Comprehension and Generation

TL;DR

We introduce JavisGPT, a multimodal LLM that can understand audiovisual inputs and simultaneously generate synchronized sounding videos in a unified model. We also curate the JavisInst-Omni dataset to facilitate instruction-tuning for comprehension and generation on sounding videos.

📰 News

• [2025.12.30] 🚀 We release the training dataset of JavisInst-Omni to support multimodal instruction tuning on sounding video comprehension and generation tasks, as well as MM-PreTrain and AV-FineTune datasets to enable preliminary multimodal alignment for LLMs. The JavisUnd-Eval dataset is also released to set a standard for audio-video understanding evaluation for MLLMs.
• [2025.12.26] 🔥 We release the code of JavisGPT, with the preview JavisGPT-v0.1-7B-Instruct checkpoint at huggingface. Feel free to play with it!

The JavisUnd-Eval Dataset

Introduction

JavisUnd-Eval is constructed to evaluate omni MLLMs for : (1) audio comprehension, (2) video comprehension, and (3) joint audio-video comprehension capabilities. The data sources mainly come from VideoLLaMA2, and we integrate the data and evaluation code to support future research in the community. For more details please refer to our paper.

Usage

First, download the dataset from huggingface:

huggingface-cli download --repo-type dataset JavisVerse/JavisUnd-Eval --local-dir /path/to/JavisUnd-Eval

Then, extract the .zip files:

cd /path/to/JavisUnd-Eval
# the raw `.zip` files will be deleted with the `--purge` flag
python unzip.py --purge

Data source and QA pairs are organized with .json files under the meta/ subfolder.

Citation

If you find JavisGPT is useful and use it in your project, please kindly cite:

@inproceedings{liu2025javisgpt,
    title={JavisGPT: A Unified Multi-modal LLM for Sounding-Video Comprehension and Generation},
    author={Kai Liu and Jungang Li and Yuchong Sun and Shengqiong Wu and jianzhang gao and Daoan Zhang and Wei Zhang and Sheng Jin and Sicheng Yu and Geng Zhan and Jiayi Ji and Fan Zhou and Liang Zheng and Shuicheng YAN and Hao Fei and Tat-Seng Chua},
    booktitle={The Thirty-ninth Annual Conference on Neural Information Processing Systems},
    year={2025},
}