A Unified Framework for Comprehensive Evaluation of Audio Foundation Models

中文 | English | 💬Discord | UltraEval-Audio Paper

🎉 We are delighted to announce that **UltraEval-Audio has been published at ACL 2026 **! Read the paper on the ACL Anthology.

v1.1 Highlights

• Popular model replication: Added replication support for popular models, including replication result showcases and one-click replication commands (see replication/).
• Isolated Runtime: Introduced an isolated inference mechanism. Model-specific dependencies are installed/managed automatically; inference runs in the isolated environment and communicates with the main evaluation process via IPC, eliminating dependency conflicts.
• Specialized model evaluation support: Added specialized audio models for TTS, ASR, and Audio Codec, further expanding evaluation coverage.

Overview

🚀Exceptional Experience with UltraEval-Audio🚀

UltraEval-Audio — The world's first open-source framework supporting both speech understanding and speech generation evaluation, specifically designed for large audio models. It aggregates 34 authoritative benchmarks, covering four major domains: speech, sound, medicine, and music, supporting 10 languages and 12 task categories. With UltraEval-Audio, you will experience unprecedented convenience and efficiency:

• Direct Replication of Popular Models 🔬: Provides detailed replication documentation and commands, ensuring you can easily reproduce evaluation results of open-source models with complete transparency and reproducibility.
• One-Click Benchmark Management 📥: Say goodbye to tedious manual downloading and data processing. UltraEval-Audio automates it all, letting you easily acquire well-known benchmark datasets (e.g., Librispeech, TED-LIUM, Seed-TTS-Eval).
• Built-in Evaluation Tools ⚙️: No need to hunt for evaluation tools. UltraEval-Audio binds datasets with commonly used official evaluation methods (e.g., WER, WER-ZH, BLEU, G-Eval) to ensure alignment between datasets and metrics.
• Powerful and Flexible 🛠️: Supports preview testing, random sampling, error retries, and resume-from-breakpoint, ensuring a flexible and controllable evaluation process while boosting efficiency and accuracy.
• Seamless Integration of Custom Datasets 💼: Supports not only public benchmarks but also powerful custom dataset integration, allowing rapid application in various engineering scenarios.
• Easy Integration with Existing Systems 🔗: With excellent extensibility and standardized design, UltraEval-Audio seamlessly connects with your existing evaluation pipelines, simplifying project management and unifying output results.

Changelog🔥

• [2026/08/25]
  • Support FireRedTTS3 evaluation (fireredtts3-base, fireredtts3-instruct), covering Seed-TTS-Eval, MiniMax multilingual TTS (24 languages), and InstructTTSEval
• [2026/08/12]
  • Support Fun-ASR-Nano-2512 evaluation (fun-asr-nano-2512) through a pinned isolated FunASR runtime.
• [2026/07/31]
  • Support MOSS-TTS-v1.5 evaluation (moss-tts-v1.5), including Seed-TTS-Eval, CV3 zero-shot, and MiniMax multilingual TTS benchmarks (22 languages)
• [2026/07/27]
  • Support VoxCPM2 evaluation (voxcpm2, voxcpm2-denoise), covering Seed-TTS-Eval, CV3-Eval zero-shot, and MiniMax multilingual TTS benchmarks
• [2026/07/13]
  • Support InstructTTSEval for evaluating complex natural-language instruction following in TTS systems.
  • Includes English and Chinese subsets and evaluates fine-grained acoustic control (APS), descriptive style following (DSD), and role-play/scenario style following (RP) with a Gemini judge.
  • Provides reference-audio alignment results and one-command evaluation examples for the integrated benchmark.
• [2026/06/10]
  • Support Qwen3-ASR evaluation (qwen3-asr-1.7b, qwen3-asr-0.6b), with replication results and commands for English, Chinese, and Chinese dialect ASR benchmarks.
• [2026/04/20]
  • Support Fish Speech S2 Pro evaluation, including Seed-TTS-Eval and MiniMax multilingual TTS benchmarks (22 languages)
• [2026/02/03]
  • Support Qwen3-TTS evaluation
  • GPU parallel acceleration for faster evaluation/inference
    • Usage: add --use_model_pool and --workers <N> to enable multi-GPU parallel inference, e.g.
      • python audio_evals/main.py --dataset <dataset_name> --model <model_name> --use_model_pool --workers 4
• [2026/01/19]
  • Support Step-Audio-R1.1 evaluation, with replication report: Step-Audio-R1.1
• [2025/12/31]
  • release v1.1 🎉🎉🎉
    • Add replication docs for popular models: CosyVoice2, CosyVoice3, GLM-TTS, IndexTTS2, VoxCPM
    • support Isolated Runtime offline inference
    • support TTS、ASR、Audio Codec specific task audio model
• [2025/12/04]
  • Support Qwen3-Omni, update Kimi-Audio
• [2025/12/02]
  • 🌟 Added Replication Results and Command Documentation: To better support the open-source community, we have detailed the evaluation process and results of current open-source models, ensuring the evaluation process is completely transparent and reproducible.
  • Support Long-TTS-Eval dataset, see alignment details in Long-TTS-Eval
  • Support MGM-Omni TTS model, see alignment details in MGM-Omni
• [2025/10/30]
  • Support VoxCPM TTS model: --model voxcpm-tts --model voxcpm-vc
  • Use uv to accelerate model dependency installation 🚀
• [2025/10/17]
  • Support seed-tts-eval dataset
• [2025/05/22]
  • Use audio quality metrics
• [2025/05/12]
  • Support Qwen2.5-Omni qwen2.5-omni-audio, qwen2.5-omni-speech, Kimi-Audio-7B-Instruct kimiaudio, kimiaudio-speech models, and update Audio Understanding Leaderboard
• [2025/05/8]
  • Faster resume evaluation, -r/--resume parameter, automatically searches for the latest breakpoint result if no file is specified
  • Support evaluation starting from inference file, --infer-file parameter, allows direct evaluation from inference file without regeneration
• [2025/03/23]
  • Added support for step-audio model evaluation and ranking
    • Ranking details: leaderboard.md
    • Evaluation support: Step-Audio-Chat
• [2025/03/04]
  • Support [resume evaluation](docs/Procedures for Restarting an Incomplete Evaluation.md), command line parameter --resume $checkpoint_res_file
  • glm-4-voice service deployment, supports UltraEval-Audio evaluation, see details at GLM-4-Voice
  • Parallel evaluation support, command line parameter --workers $num_workers
• [2025/01/13] release v1.0

Leaderboard

Audio Understanding Leaderboard

Audio Understanding Audio Foundation Models: Speech + Text → Text

WER/CER ($\downarrow$) for ASR, BLEU ($\uparrow$) for AST, and ACC ($\uparrow$) for EMO. Best results are in bold.

Scoring:

• Avg. Score ($\uparrow$): mean of all available normalized metric scores. For WER/CER-based metrics we use ((100-\text{WER/CER})); for other metrics (e.g., BLEU/Acc.) we keep the original value.

Audio Generation Leaderboard

Audio Generation Audio Foundation Models: Speech → Speech Table: Audio generation performance ($\uparrow$). Acoustic metrics (UTMOS | DNSMOS P.835 | DNSMOS P.808, scores range from 0 to 5) are evaluated on the generated audio responses from the speech tasks. Best results are in bold.

Note: The average score is computed as the average of 6 scores: five speech-task scores and normalized acoustic scores. For acoustic scores (UTMOS | DNSMOS P.835 | DNSMOS P.808), each value (0--5) is multiplied by 20 to map to 0--100, then averaged to obtain the normalized acoustic score.

Audio Codec Leaderboard

Audio Codec: Speech → Speech. Table: Audio Codec Performance: ASR-WER ($\downarrow$), ASR-CER ($\downarrow$), SIM ($\uparrow$), and Acoustics (UTMOS|DNSMOS P.835|DNSMOS P.808, $\uparrow$). Note: The hyphen (-) indicates that UTMOS is not applicable to Chinese speech (AISHELL-1). Best results are in bold.

Note: For acoustic scores we use UTMOS, DNSMOS P.835, and DNSMOS P.808 metrics. To calculate the average score, for ASR-WER and ASR-CER, we calculate (100-\text{val}). For acoustic scores, each available value (ranges from 0 to 5) is normalized by (20\times\mathrm{val}) (mapping to 0--100), and the acoustic score is their average (the hyphen `-' is ignored). The final score is the average of 9 metric scores.

Quick Start

Environment Preparation

git clone https://github.com/OpenBMB/UltraEval-Audio.git
cd UltraEval-Audio
conda create -n env python=3.10 -y
conda activate env
pip install -e .

or use uv for faster installation:

uv venv env --python 3.10
source env/bin/activate
uv pip install -e .

Run Examples

# For some regions, you may need to set: export HF_ENDPOINT=https://hf-mirror.com
# Test MiniCPM-o 2.6 speech understanding capability
CUDA_VISIBLE_DEVICES=0 python audio_evals/main.py --dataset sample --prompt mini-cpm-omni-asr-zh --model MiniCPMo2_6-audio

# Test MiniCPM-o 2.6 speech generation capability
CUDA_VISIBLE_DEVICES=0 python audio_evals/main.py --dataset llama-questions-s2t --model MiniCPMo2_6-speech

# Test GPT-4o-Realtime speech understanding capability
export OPENAI_API_KEY=$your-key
python audio_evals/main.py --dataset sample --model gpt4o_audio

# Test GPT-4o-Realtime speech generation capability
export OPENAI_API_KEY=$your-key
python audio_evals/main.py --dataset llama-questions-s2t --model gpt4o_speech

# Test gemini-1.5-pro speech understanding capability
export GOOGLE_API_KEY=$your-key
python audio_evals/main.py --dataset sample --model gemini-pro

# Test Qwen3-ASR speech recognition capability
CUDA_VISIBLE_DEVICES=0 python audio_evals/main.py --dataset librispeech-test-clean --model qwen3-asr-1.7b --prompt simple-asr
# See full replication results and commands: replication/qwen3_asr.md

# Test Fun-ASR-Nano multilingual speech recognition capability
CUDA_VISIBLE_DEVICES=0 python audio_evals/main.py --dataset librispeech-test-clean --model fun-asr-nano-2512 --prompt simple-asr

# Test qwen2-audio-offline speech understanding capability
CUDA_VISIBLE_DEVICES=0 python audio_evals/main.py --dataset sample --model qwen2-audio-chat

If you encounter errors or cannot reproduce Mini-CPM-o 2.6 results, please check FAQ.

Res

Evaluation complete, results are as follows:

- res
    |-- $model-name
        |-- $dataset
            |-- $time.jsonl
            |-- $time-overview.jsonl

Usage

Evaluation command:

python audio_evals/main.py --dataset <dataset_name> --model <model_name>

Dataset Selection

<dataset_name> specifies the dataset to evaluate. Supported datasets can be viewed via python cli/list_availabel.py

Construct your own dataset: docs/how add a dataset.md

Model Selection

model_name specifies the model to evaluate. Supported models can be viewed via python cli/list_availabel.py

Evaluate your own model docs/how eval your model.md

Contact Us

If you have any suggestions or questions, please file an issue or join our Discord group: https://discord.com/invite/Qrsbft4e

Citation

If you find UltraEval-Audio helpful, please consider citing our paper 📝 and staring us ⭐️！

@inproceedings{shi-etal-2026-ultraeval,
    title = "{U}ltra{E}val-Audio: A Unified Framework for Comprehensive Evaluation of Audio Foundation Models",
    author = "Shi, Qundong  and
      Zhou, Jie  and
      Lin, Biyuan  and
      Cui, Junbo  and
      Zeng, Guoyang  and
      Zhou, Yixuan  and
      Wang, Ziyang  and
      Liu, Xin  and
      Luo, Zhen  and
      Wang, Yudong  and
      Liu, Zhiyuan",
    editor = "Durrett, Greg  and
      Jian, Ping",
    booktitle = "Proceedings of the 64th Annual Meeting of the {A}ssociation for {C}omputational {L}inguistics (Volume 3: System Demonstrations)",
    month = jul,
    year = "2026",
    address = "San Diego, California, United States",
    publisher = "Association for Computational Linguistics",
    url = "https://aclanthology.org/2026.acl-demo.56/",
    doi = "10.18653/v1/2026.acl-demo.56",
    pages = "566--577",
    ISBN = "979-8-89176-392-0",
}