---

💡 Training-Free Omni

Training-Free Omni equips frozen vision-language models with speech-centric omni understanding without model training. It converts speech into structured textual evidence and combines it with image or video inputs through a unified evaluation pipeline.

---

🔥 Highlights

• Training-free: extends frozen VLMs to speech-centric omni understanding without fine-tuning.
• Unified multimodal evaluation: supports text, image, video, audio, and audio-visual tasks.
• Broad model coverage: includes Qwen, Gemma, MiniCPM, VILA, and OmniVinci adapters.
• Broad benchmark coverage: provides 118 dataset adapters with shared evaluation and reporting.
• Optional spoken output: combines multimodal answers with CosyVoice3 speech synthesis.

---

🚀 Quick Start

Installation

Python 3.10+ and ffmpeg are recommended. Install the correct PyTorch build for your CPU, CUDA, or ROCm platform first.

python3 -m venv .venv
source .venv/bin/activate
python3 -m pip install --upgrade pip
python3 -m pip install -r requirements.txt

The root requirements.txt is consolidated. Some model adapters retain their own environment instructions; follow the selected adapter's README when provided.

Run an Evaluation

Run all commands from the repository root:

MODEL=qwen25omni \
MODEL_PATH=Qwen/Qwen2.5-Omni-3B \
EVAL_DATASETS=daily_omni \
DATA_DIR=data/daily_omni \
OUTPUT_DIR=results/qwen25omni \
LLM_JUDGE=False \
bash eval.sh

Use comma-separated names to evaluate multiple datasets:

EVAL_DATASETS="daily_omni,omnibench,av_odyssey" \
MODEL=qwen25omni \
MODEL_PATH=Qwen/Qwen2.5-Omni-3B \
OUTPUT_DIR=results/qwen25omni \
bash eval.sh

For a short smoke test, add MAX_SAMPLES=3. Set RESUME=True to continue an interrupted run.

Model Configurations

Use the evaluation command above and replace MODEL and MODEL_PATH with one of the following configurations.

Model IDs may be replaced with local checkpoint paths for offline evaluation.

Direct Python Evaluation

python3 run_eval.py \
  --model qwen25omni \
  --model_path Qwen/Qwen2.5-Omni-3B \
  --dataset daily_omni \
  --data_dir data/daily_omni \
  --output_dir results/qwen25omni \
  --llm_judge False

End-to-End Text and Speech Output

Prepare CosyVoice3 once:

bash End_to_End_Model/setup_cosyvoice3.sh

Then run multimodal inference with text and spoken output:

python3 End_to_End_Model/run_qwen25vlomni_advance.py \
  --video path/to/video.mp4 \
  --audio path/to/audio.wav \
  --question "What happens in this video?" \
  --output-mode both \
  --output-audio outputs/answer.wav

---

📊 Evaluation Outputs

Each run writes:

<OUTPUT_DIR>/<dataset>/predictions.jsonl
<OUTPUT_DIR>/<dataset>/summary.json
<OUTPUT_DIR>/all_results.csv

Dataset names correspond to folders under datasets/. Model names correspond to folders under models/. Model checkpoints and benchmark data are not included in this repository.

---

📜 Citation

@article{deria2026training,
  title={Training-Free Speech-Centric Omni Understanding with Frozen VLMs},
  author={Deria, Ankan and Rasheed, Hanoona and He, Xilin and Khan, Fahad Shahbaz and Khan, Salman},
  journal={arXiv preprint arXiv:2609.04242},
  year={2026}
}

---