WAVE: Learning Unified & Versatile Audio-Visual Embeddings with Multimodal LLM

---

🔥 News

• [2026/07/30]: 💻 WAVE can now be loaded directly via Hugging Face transformers!
• [2026/02/11]: 🚀 We released the official code and model checkpoints!
• [2026/02/06]: 🎉 WAVE has been accepted to ICLR 2026 as an Oral Presentation!

---

📖 Abstract

We introduce WAVE (Unified & Versatile Audio-Visual Embeddings), the first LLM-based embedding model that creates a unified representation space for text, audio, silent video, and synchronized audio-visual modalities.

Unlike previous methods that use separate encoders, WAVE is built upon Qwen2.5-Omni and employs a novel hierarchical feature fusion strategy. It features a dual-encoder design for audio (Speech + Environmental Sounds) and is trained via a joint multi-modal, multi-task approach.

Key Capabilities:

1. Any-to-Any Retrieval: Seamlessly retrieve across text, audio, and video modalities.
2. Prompt-Aware Embeddings: Generate embeddings tailored to user instructions (e.g., specific questions), significantly boosting performance in multimodal QA.
3. SOTA Performance: WAVE achieves state-of-the-art results on the MMEB-v2-video benchmark and excels in audio/audio-visual retrieval tasks.

---

🏆 Performance

WAVE achieves SOTA performance on MMEB-v2-video, outperforming industrial-grade models like Seed-1.6-Embedding.

WAVE also demonstrates strong performance across audio and audio-visual modalities.

🛠️ Installation

For key library versions, please refer to requirements.txt. Install via:

pip install -r requirements.txt

---

🤖 Model Zoo

We provide the model checkpoint on Hugging Face.

---

🚀 Quick Start

1. Basic Usage (Hugging Face Transformers)

We have implemented a lightweight usage method. You can load and use WAVE directly using the Hugging Face transformers library:

import torch
from transformers import AutoModel, AutoTokenizer

# Load the model and tokenizer
model_id = "tsinghua-ee/WAVE-7B"
tokenizer = AutoTokenizer.from_pretrained(model_id, trust_remote_code=True)
model = AutoModel.from_pretrained(
    model_id, 
    trust_remote_code=True, 
    torch_dtype=torch.bfloat16
).cuda()

2. Benchmark Evaluation

If you want to evaluate WAVE using our code, you should follow these steps:

1. Download Models: including the WAVE checkpoint and the BEATs checkpoint. Modify the path in train_qwen.py: https://github.com/TCL606/WAVE/blob/main/qwenvl/train/train_qwen.py#L184, https://github.com/TCL606/WAVE/blob/main/qwenvl/train/train_qwen.py#L269

2. Data Preparation: Organize your data into the sample JSON format. Refer to the templates provided in the scripts/ directory, such as:

   • scripts/ret_av2t.json (video/audio to text retrieval)
   • scripts/ret_mc.json (QA)
   • scripts/ret_v2a.json (audio-visual retrieval)

3. Inference: Modify scripts/direct_test.sh, then run the script:

   bash scripts/direct_test.sh
   

   The results will be automatically saved in the output/test directory.

---

📝 Citation

If you find WAVE useful for your research, please consider citing our paper:

@inproceedings{tang2026wave,
    title={{WAVE}: Learning Unified \& Versatile Audio-Visual Embeddings with Multimodal {LLM}},
    author={Changli Tang and Qinfan Xiao and Ke Mei and Tianyi Wang and Fengyun Rao and Chao Zhang},
    booktitle={The Fourteenth International Conference on Learning Representations},
    year={2026},
    url={https://openreview.net/forum?id=MiV3WXDYJb}
}

---

License: Apache 2.0