Enhance Generation Quality of Flow Matching V2A Model via Multi-Step CoT-Like Guidance and Combined Preference Optimization

Haomin Zhang, Sizhe Shan, Haoyu Wang, Zihao Chen, Xiulong Liu, Chaofan Ding, Xinhan Di

AI Lab Giant Network, Zhejiang University, University of Washington

Results

Objective metrics

Results of Video-to-Audio Synthesis

https://github.com/user-attachments/assets/d6761371-8fc2-427c-8b2b-6d2ac22a2db2

https://github.com/user-attachments/assets/50b33e54-8ba1-4fab-89d3-5a5cc4c22c9a

Results of Video-to-Piano Synthesis

https://github.com/user-attachments/assets/1bf7cb2e-084a-4c70-969d-4da6a9632385

https://github.com/user-attachments/assets/040cd7bd-e74c-4b06-ace0-afba2271fa49

https://github.com/user-attachments/assets/ddf762ad-2d12-41b2-b3aa-edf0d8f9549c

https://github.com/user-attachments/assets/5a6baa3d-f563-4812-93c9-2faad3af9cba

https://github.com/user-attachments/assets/d06a7afc-4b8f-4b4d-bf27-29624355da03

https://github.com/user-attachments/assets/7a3cd7fc-6a27-476a-8b24-aa5c8ddbfff2

https://github.com/user-attachments/assets/2556e012-bbc9-465a-a107-16d77761ae89

https://github.com/user-attachments/assets/be14f825-0133-490d-bc20-5b760ffc8588

Installation

1. Create a conda environment

conda create -n v2ap python=3.10
conda activate v2ap

2. Install requirements

pip install -r requirements.txt

Pretrained models

The models are available at https://huggingface.co/lshzhm/Video-to-Audio-and-Piano/tree/main.

Inference

1. Video-to-Audio inference

python src/inference_v2a.py

2. Video-to-Piano inference

python src/inference_v2p.py

Dateset is in progress

Acknowledgement

• Audeo for video to midi prediction
• E2TTS for CFM structure and base E2 implementation
• FLAN-T5 for FLAN-T5 text encode
• CLIP for CLIP image encode
• AudioLDM Eval for audio evaluation