MOVA: Towards Scalable and Synchronized Video–Audio Generation

MOVA (MOSS Video and Audio) generates video and synchronized audio in a single model. This repository provides model weights, inference, training, LoRA fine-tuning, and evaluation workflows.

Start here: Quick start · Watch samples · Model weights · LoRA fine-tuning · Evaluation

🌟Key Highlights

• Native Bimodal Generation: Moves beyond clunky cascaded pipelines. MOVA generates high-fidelity video and synchronized audio in a single inference pass, eliminating error accumulation.
• Precise Lip-Sync & Sound FX: Achieves state-of-the-art performance in multilingual lip-synchronization and environment-aware sound effects.
• Fully Open-Source: In a field dominated by closed-source models (Sora 2, Veo 3, Kling), we are releasing model weights, inference code, training pipelines, and LoRA fine-tuning scripts.
• Asymmetric Dual-Tower Architecture: Leverages the power of pre-trained video and audio towers, fused via a bidirectional cross-attention mechanism for rich modality interaction.

🔥News!!!

• 2026/05/06: 🎉We released the evaluation code for MOVA! See evaluation/ for details.
• 2026/04/01: 🎉We updated the MOVA Benchmark for Arena with videos generated by different MOVA variants for Arena-based evaluation.
• 2026/03/14: 🎉We released the MOVA Benchmark for Arena (732 samples) on 🤗 Hugging Face for reproducible evaluation.

Earlier updates

• 2026/03/09: 🎉MOVA API is now available! Apply for your API key at studio.mosi.cn to start generating videos programmatically.
• 2026/03/09: 🎉ComfyUI support is here! Thanks to @richservo, you can now use MOVA in ComfyUI at low cost via comfyui-mova.
• 2026/02/10: 🎉We released MOVA technical report and update inference workflow.
• 2026/01/29: 🎉We released MOVA, an open-source foundation model for high-fidelity synchronized video–audio generation!!!

🎬Demo

Single person speech:

Click to expand

Multi-person speech:

Click to expand

View more demos on our website.

🚀Quick Start

Environment Setup

conda create -n mova python=3.13 -y
conda activate mova
pip install -e .

Model Downloading

hf download OpenMOSS-Team/MOVA-360p --local-dir /path/to/MOVA-360p
hf download OpenMOSS-Team/MOVA-720p --local-dir /path/to/MOVA-720p

Inference

Generate a video of single person speech:

export CP_SIZE=1
export CKPT_PATH=/path/to/MOVA-360p/

torchrun \
    --nproc_per_node=$CP_SIZE \
    scripts/inference_single.py \
    --ckpt_path $CKPT_PATH \
    --cp_size $CP_SIZE \
    --height 352 \
    --width 640 \
    --prompt "A man in a blue blazer and glasses speaks in a formal indoor setting, framed by wooden furniture and a filled bookshelf. Quiet room acoustics underscore his measured tone as he delivers his remarks. At one point, he says, \"I would also say that this election in Germany wasn’t surprising.\"" \
    --ref_path "./assets/single_person.jpg" \
    --output_path "./data/samples/single_person.mp4" \
    --seed 42 \
    --offload cpu

Generate a video of multi-person speech:

export CP_SIZE=1
export CKPT_PATH=/path/to/MOVA-360p/

torchrun \
    --nproc_per_node=$CP_SIZE \
    scripts/inference_single.py \
    --ckpt_path $CKPT_PATH \
    --cp_size $CP_SIZE \
    --height 352 \
    --width 640 \
    --prompt "The scene shows a man and a child walking together through a park, surrounded by open greenery and a calm, everyday atmosphere. As they stroll side by side, the man turns his head toward the child and asks with mild curiosity, in English, \"What do you want to do when you grow up?\" The boy answers with clear confidence, saying, \"A bond trader. That's what Don does, and he took me to his office.\" The man lets out a soft chuckle, then responds warmly, \"It's a good profession.\" as their walk continues at an unhurried pace, the conversation settling into a quiet, reflective moment." \
    --ref_path "./assets/multi_person.png" \
    --output_path "./data/samples/multi_person.mp4" \
    --seed 42 \
    --offload cpu

Please refer to the inference script for more argument usage.

Key optional arguments (scripts/inference_single.py)

--offload cpu: component-wise CPU offload to reduce VRAM, typically slower and uses more Host RAM.
--offload group: finer-grained layerwise/group offload, often achieves lower VRAM but is usually slower and increases Host RAM pressure (see the benchmark table below).
--remove_video_dit: after switching to low-noise video_dit_2, frees the stage-1 video_dit reference, which can reduce ~28GB of Host RAM when offload is enabled.

Inference Performance Reference

We provide inference benchmarks for generating an 8-second 360p videos under different offloading strategies. Note that actual performance may vary depending on hardware configurations, driver versions, and PyTorch/CUDA builds.

Ascend NPU support

We also support NPUs. For more details about NPU training/inference, please refer to this document.

Evaluation

We release our evaluation code in the evaluation/ directory, which provides 11 metrics across 7 metric groups for assessing synchronized audio-visual video generation quality, including audio quality, lip-sync, AV alignment, and more.

We evaluate our model through both objective benchmarks and subjective human evaluations.

Evaluation on Verse-Bench

We provide quantitative comparison of audiovisual generation performance on Verse-Bench. The Audio and AV-Align metrics are evaluated on all subsets; the Lip Sync and Speech metrics are evaluated on Verse-Bench Set3; and ASR Acc is evaluated on a multi-speaker subset proposed by our team. Boldface and underlined numbers indicate the best and second-best results, respectively.

In the lip-sync task, which shows the largest performance gap, MOVA demonstrates a clear advantage. According to the Lip Sync Error metric, with Dual CFG enabled, MOVA-720p achieves an LSE-D score of 7.094 and an LSE-C score of 7.452. Furthermore, MOVA also attains the best performance on the cpCER metric, which reflects speech recognition accuracy and speaker-switching accuracy.

Human Evaluation

Below are the Elo scores and win rates comparing MOVA to existing open-source models.

MOVA Benchmark for Arena

We release the MOVA Benchmark for Arena on Hugging Face for reproducible subjective evaluation. The benchmark contains 732 samples organized into two subsets:

Each sample includes a first-frame image and a prompt (rewritten by the workflow introduced in the paper) for joint image-text to video-audio generation.

🤗 Download: zhiyuzhang-0212/MOVA_benchmark_for_arena

SGLang Integration

SGLang provides Day0-support for MOVA. You can use the latest SGLang release and the examples below for high-throughput inference.

CLI Generation (sglang generate)

sglang generate \
  --model-path OpenMOSS-Team/MOVA-720p \
  --prompt "A man in a blue blazer and glasses speaks in a formal indoor setting, \
  framed by wooden furniture and a filled bookshelf. \
  Quiet room acoustics underscore his measured tone as he delivers his remarks. \
  At one point, he says, \"I would also believe that this advance in AI recently wasn’t unexpected.\"" \
  --image-path "./assets/single_person.jpg" \
  --adjust-frames false \
  --num-gpus 8 \
  --ring-degree 2 \
  --ulysses-degree 4 \
  --num-frames 193 \
  --fps 24 \
  --seed 67 \
  --num-inference-steps 25 \
  --enable-torch-compile \
  --save-output

Online Serving (sglang serve)

export SG_OUTPUT_DIR=/root/output_mova
mkdir -p "$SG_OUTPUT_DIR"

sglang serve \
  --model-path OpenMOSS-Team/MOVA-720p \
  --host 0.0.0.0 \
  --port 30002 \
  --adjust-frames false \
  --num-gpus 8 \
  --ring-degree 2 \
  --ulysses-degree 4 \
  --tp 1 \
  --enable-torch-compile \
  --save-output \
  --output-dir "$SG_OUTPUT_DIR"

API Request Example (/v1/videos)

curl -X POST "http://0.0.0.0:30002/v1/videos" \
  -F "prompt=A man in a blue blazer and glasses speaks in a formal indoor setting, framed by wooden furniture and a filled bookshelf. Quiet room acoustics underscore his measured tone as he delivers his remarks. At one point, he says, \"I would also believe that this advance in AI recently wasn't unexpected.\"" \
  -F "input_reference=@./assets/single_person.jpg" \
  -F "size=640x352" \
  -F "num_frames=193" \
  -F "fps=24" \
  -F "seed=67" \
  -F "guidance_scale=5.0" \
  -F "num_inference_steps=25" \
  -o create_video.json

Generation Workflow

We provide an AI-assisted video generation workflow for end-to-end video creation via a Streamlit web UI. It supports:

• Full Workflow Mode: Describe a scene in text (optionally upload a first frame) → AI generates first frame if needed → extracts visual elements → rewrites into video description → MOVA generates video
• Simple Mode: Provide prompt + first frame image directly → MOVA generates video

See the workflow README for setup, configuration (SGLang server, API keys), and usage.

API Access

MOVA is available as a hosted API on our platform. You can apply for an API key and start generating synchronized video-audio content without managing any infrastructure.

Apply here: studio.mosi.cn/docs/models/mova

ComfyUI Integration

MOVA is supported in ComfyUI through the community plugin comfyui-mova by @richservo. It provides two nodes — MOVA Model Loader and MOVA Generate — for both Image-to-Video-Audio (I2VA) and Text-to-Video-Audio (T2VA) workflows.

Installation

cd ComfyUI/custom_nodes
git clone --recursive https://github.com/richservo/comfyui-mova.git
cd comfyui-mova
pip install -r requirements.txt

Models download automatically from HuggingFace on first use. See the comfyui-mova README for full usage details, VRAM guide, and workflow examples.

Training

LoRA Fine-tuning

The following commands show how to launch LoRA training in different modes; for detailed memory and performance numbers, see the LoRA Resource & Performance Reference section below.

Training Preparation

• Model checkpoints: Download MOVA weights to your local path and update the diffusion_pipeline section of the corresponding config.
• Dataset: Configure your video+audio dataset and transforms in the data section of the corresponding config (e.g., mova_train_low_resource.py); see mova/datasets/video_audio_dataset.py for the expected fields.
• Environment: Use the same environment as inference, then install training-only extras: pip install -e ".[train]" (includes torchcodec and bitsandbytes).
• Configs: Choose one of the training configs below and edit LoRA, optimizer, and scheduler settings as needed.

Low-resource LoRA (single GPU, most memory-efficient)

• Config: configs/training/mova_train_low_resource.py
• Script:

bash scripts/training_scripts/example/low_resource_train.sh

Accelerate LoRA (1 GPU)

• Config: configs/training/mova_train_accelerate.py
• Script:

bash scripts/training_scripts/example/accelerate_train.sh

Accelerate + FSDP LoRA (8 GPUs)

• Config: configs/training/mova_train_accelerate_8gpu.py
• Accelerate config: configs/training/accelerate/fsdp_8gpu.yaml
• Script:

bash scripts/training_scripts/example/accelerate_train_8gpu.sh

All hyper-parameters (LoRA rank/alpha, target modules, optimizer, offload strategy, etc.) are defined in the corresponding config files; the example scripts only take the config path as input.

LoRA Resource & Performance Reference

All peak usage numbers below are measured on 360p, 8-second video training settings and will vary with resolution, duration, and batch size.

Note: Training 8-second 360p videos on RTX 4090 is not recommended due to high resource requirements and slow training speed. We strongly suggest reducing video resolution (e.g., 240p) or total frame count to accelerate training and reduce resource consumption.

📑TODO List

• Checkpoints
• Multi-GPU inference
• Lora fine-tune
• Ascend NPU Fine-tune
• Ascend NPU Inference
• SGLang Integration
• Generation Workflow
• Technical Report
• API Access
• ComfyUI Integration
• Arena Benchmark
• Evaluation Code
• Diffusers Integration

Citation

If you find our work helpful, please cite us.

@misc{openmoss_mova_2026,
  title         = {MOVA: Towards Scalable and Synchronized Video-Audio Generation},
  author        = {{SII-OpenMOSS Team} and Donghua Yu and Mingshu Chen and Qi Chen and Qi Luo and Qianyi Wu and Qinyuan Cheng and Ruixiao Li and Tianyi Liang and Wenbo Zhang and Wenming Tu and Xiangyu Peng and Yang Gao and Yanru Huo and Ying Zhu and Yinze Luo and Yiyang Zhang and Yuerong Song and Zhe Xu and Zhiyu Zhang and Chenchen Yang and Cheng Chang and Chushu Zhou and Hanfu Chen and Hongnan Ma and Jiaxi Li and Jingqi Tong and Junxi Liu and Ke Chen and Shimin Li and Songlin Wang and Wei Jiang and Zhaoye Fei and Zhiyuan Ning and Chunguo Li and Chenhui Li and Ziwei He and Zengfeng Huang and Xie Chen and Xipeng Qiu},
  year          = {2026},
  month         = feb,
  eprint        = {2602.08794},
  archivePrefix = {arXiv},
  primaryClass  = {cs.CV},
  doi           = {10.48550/arXiv.2602.08794},
  url           = {https://arxiv.org/abs/2602.08794},
  note          = {Technical report. Corresponding authors: Xie Chen and Xipeng Qiu. Project leaders: Qinyuan Cheng and Tianyi Liang.}
}

Acknowledgement

We would like to thank the contributors to Wan, SGLang, diffusers, HuggingFace, DiffSynth-Studio, and HunyuanVideo-Foley for their great open-source work, which is helpful to this project.

Star History