Ola-Omni/Ola

Ola: Pushing the Frontiers of Omni-Modal Language Model

Python

395

29 commits

updated Jun 13, 2025

ola-omni.github.io

See the code

README

Zuyan Liu^*,1,2 Yuhao Dong^*,2,3 Jiahui Wang¹
Ziwei Liu³ Winston Hu² Jiwen Lu^1,✉ Yongming Rao^2,1,✉

¹Tsinghua University ²Tencent Hunyuan Research ³S-Lab, NTU

^* Equal Contribution^✉ Corresponding Author

Project Page:

Weights in Huggingface:

arXiv Paper:

Demo by Gradio:

Training Data:

中文解读:

Contact: Leave an issue or contact liuzuyan19@gmail.com . We are on call to respond.

📢 News

🔥[28/2/2025] We release the intermediate model, Ola-Image and Ola-Video, try building your own omni-modal models!
🚀[19/2/2025] We release the huggingface demo of Ola, try the advanced omni-modal model on your own!
🔥[18/2/2025] The training data, training script for Ola-7b is released!
🎉[07/2/2025] The Ola is released! Check our project page, model weights, arXiv paper for the strong omni-modal understanding model!
🔥[06/2/2025] Ola-7b achieves Rank #1 on the OpenCompass Multi-modal Leaderboard among all the models under 15B parameters with average score of 72.6. Check the impressive results here!

🚀Coming Soon

Evaluation code on omni-modal benchmarks
Gradio Demo
Training Data (Video, Audio, Cross-Modality)

🌟 Introduction

Roads to Ola

Ola is an Omni-modal language model that achieves competitive performance across image, video, and audio understanding compared to specialized counterparts. We conduct a comprehensive exploration of architectural design, data curation, and training strategies essential for building a robust omni-modal model.

Architecture

Ola supports omni-modal inputs including text, image, video, and audio, capable of processing the inputs simultaneously with competitive performance on understanding tasks for all these modalities. Meanwhile, Ola supports user-friendly real-time streaming decoding for texts and speeches thanks to the text detokenizer and the speech decoder.

Training Strategies

We visualize the relationships among modalities in the left part. Speech acts as the connection between language and audio knowledge, while video constructs the bridge with highly relevant visual and audio information. Therefore, we design the progressive alignment training strategy from primary to periphery. Furthermore, we design the cross-modality video-audio data to better capture the relationships among modalities.

Performance

Ola achieves competitive performance across major multi-modal benchmarks when compared to state-of-the-art specialist-modal LLMs.

Installation

1. Clone this repository:

git clone https://github.com/Ola-Omni/Ola
cd Ola

2. Install the required package:

conda create -n ola python=3.10 -y
conda activate ola
pip install --upgrade pip
pip install -e .

3.Install additional packages for training cases

pip install -e ".[train]"
pip install flash-attn --no-build-isolation

Model Zoo

We provide our checkpoints at Huggingface

Model	Link	Size	Modal
Ola-7b	Huggingface	7B	Text, Image, Video, Audio
Ola-Image	Huggingface	7B	Text, Image
Ola-Video	Huggingface	7B	Text, Image, Video

Quick Start

Download Ola-7b from Huggingface or skip the step to using the online weights directly.
Download audio encoder from Huggingface and put the weights large-v3.pt and BEATs_iter3_plus_AS2M_finetuned_on_AS2M_cpt2.pt under repo directory path/to/Ola/
Run inference/infer.py

Text & Image Understanding

python3 inference/infer.py --image_path *.png,jpg --text user_instruction

Text & Video Understanding

python3 inference/infer.py --video_path *.mp4 --text user_instruction

Text & Audio Understanding

python3 inference/infer.py --audio_path *.wav,mp3 --text user_instruction

Audio & Image Understanding

python3 inference/infer.py --audio_path *.png,jpg --audio_path *.wav,mp3

Evaluation

You can evaluate Ola model with VLMEvalKit and lmms-eval.

Training

Data Preparation

Please refer to DATA.md for instructions of customized finetuning or using the provided datasets.

Start Training

Please follow the script below to start training. Make sure you have created the correct datasets for fine-tuning.

Finetuning Ola-7b Model:

bash ./scripts/finetune_ola.sh

Finetuning Ola-Image Model (Ola Stage1 or Stage2)

bash ./scripts/finetune_ola_image.sh

Finetuning Ola-Video Model (Ola Stage3):

bash ./scripts/finetune_ola_video.sh

Citation

If you find it useful for your research and applications, please cite our paper using this BibTeX:

@article{liu2025ola,
title={Ola: Pushing the Frontiers of Omni-Modal Language Model with Progressive Modality Alignment},
author={Liu, Zuyan and Dong, Yuhao and Wang, Jiahui and Liu, Ziwei and Hu, Winston and Lu, Jiwen and Rao, Yongming},
journal={arXiv preprint arXiv:2502.04328},
year={2025}
}

Acknowledgement

Our codebase is conducted on LLaVA
Thanks VLMEvalKit and lmms-eval team for the evaluation system!

Contributors

25 commits

2 commits

1 commits

1 commits

Similar repos

THUdyh/Ola-Image

Model

Ola-7B

THUdyh/Ola-7b

Model

Ola-7B

THUdyh/Ola-Video

Model

Ola-7B

NVlabs/OmniVinci

OmniVinci is an omni-modal LLM for joint understanding of vision, audio, and language.

682

HumanMLLM/HumanOmni

HumanOmni

241

emova-ollm/EMOVA

Official PyTorch implementation of EMOVA in CVPR 2025 (https://arxiv.org/abs/2409.18042)

Meng-zzzZ/AVProunRLForVideoLLaMa2

DAMO-NLP-SG/VideoLLaMA2

VideoLLaMA 2: Advancing Spatial-Temporal Modeling and Audio Understanding in Video-LLMs

1,307

Languages

Python

98.3%

Shell

1.7%

Ola-Omni/Ola

Ola: Pushing the Frontiers of Omni-Modal Language Model

Python

395

29 commits

updated Jun 13, 2025

ola-omni.github.io

See the code

README

Zuyan Liu^*,1,2 Yuhao Dong^*,2,3 Jiahui Wang¹
Ziwei Liu³ Winston Hu² Jiwen Lu^1,✉ Yongming Rao^2,1,✉

¹Tsinghua University ²Tencent Hunyuan Research ³S-Lab, NTU

^* Equal Contribution^✉ Corresponding Author

Project Page:

Weights in Huggingface:

arXiv Paper:

Demo by Gradio:

Training Data:

中文解读:

Contact: Leave an issue or contact liuzuyan19@gmail.com . We are on call to respond.

📢 News

🔥[28/2/2025] We release the intermediate model, Ola-Image and Ola-Video, try building your own omni-modal models!
🚀[19/2/2025] We release the huggingface demo of Ola, try the advanced omni-modal model on your own!
🔥[18/2/2025] The training data, training script for Ola-7b is released!
🎉[07/2/2025] The Ola is released! Check our project page, model weights, arXiv paper for the strong omni-modal understanding model!
🔥[06/2/2025] Ola-7b achieves Rank #1 on the OpenCompass Multi-modal Leaderboard among all the models under 15B parameters with average score of 72.6. Check the impressive results here!

🚀Coming Soon

Evaluation code on omni-modal benchmarks
Gradio Demo
Training Data (Video, Audio, Cross-Modality)

🌟 Introduction

Roads to Ola

Architecture

Training Strategies

Performance

Ola achieves competitive performance across major multi-modal benchmarks when compared to state-of-the-art specialist-modal LLMs.

Installation

1. Clone this repository:

git clone https://github.com/Ola-Omni/Ola
cd Ola

2. Install the required package:

conda create -n ola python=3.10 -y
conda activate ola
pip install --upgrade pip
pip install -e .

3.Install additional packages for training cases

pip install -e ".[train]"
pip install flash-attn --no-build-isolation

Model Zoo

We provide our checkpoints at Huggingface

Model	Link	Size	Modal
Ola-7b	Huggingface	7B	Text, Image, Video, Audio
Ola-Image	Huggingface	7B	Text, Image
Ola-Video	Huggingface	7B	Text, Image, Video

Quick Start

Download Ola-7b from Huggingface or skip the step to using the online weights directly.
Download audio encoder from Huggingface and put the weights large-v3.pt and BEATs_iter3_plus_AS2M_finetuned_on_AS2M_cpt2.pt under repo directory path/to/Ola/
Run inference/infer.py

Text & Image Understanding

python3 inference/infer.py --image_path *.png,jpg --text user_instruction

Text & Video Understanding

python3 inference/infer.py --video_path *.mp4 --text user_instruction

Text & Audio Understanding

python3 inference/infer.py --audio_path *.wav,mp3 --text user_instruction

Audio & Image Understanding

python3 inference/infer.py --audio_path *.png,jpg --audio_path *.wav,mp3

Evaluation

You can evaluate Ola model with VLMEvalKit and lmms-eval.

Training

Data Preparation

Please refer to DATA.md for instructions of customized finetuning or using the provided datasets.

Start Training

Please follow the script below to start training. Make sure you have created the correct datasets for fine-tuning.

Finetuning Ola-7b Model:

bash ./scripts/finetune_ola.sh

Finetuning Ola-Image Model (Ola Stage1 or Stage2)

bash ./scripts/finetune_ola_image.sh

Finetuning Ola-Video Model (Ola Stage3):

bash ./scripts/finetune_ola_video.sh

Citation

If you find it useful for your research and applications, please cite our paper using this BibTeX:

@article{liu2025ola,
title={Ola: Pushing the Frontiers of Omni-Modal Language Model with Progressive Modality Alignment},
author={Liu, Zuyan and Dong, Yuhao and Wang, Jiahui and Liu, Ziwei and Hu, Winston and Lu, Jiwen and Rao, Yongming},
journal={arXiv preprint arXiv:2502.04328},
year={2025}
}