WavBench: Benchmarking Reasoning, Colloquialism, and Paralinguistics for End-to-End Spoken Dialogue Models

📖 Paper | 🏠 Website

Overview of WavBench

News

• 2026.02.11 Released the WavBench paper, code, and dataset.
• 2026.02.11 Released the leaderboard evaluating 5 state-of-the-art E2E spoken dialogue models.

Table of Contents

• Leaderboard
• Setup
• Dataset
• Evaluation
• Citation

Leaderboard

Below is the overall evaluation of WavBench across five panels: Colloquial Expression (Pro & Basic) and Acoustic Interaction (Explicit Understanding, Explicit Generation, and Implicit).

Setup

conda create -n wavbench python=3.10
conda activate wavbench
pip install torch==2.6.0 torchvision==0.21.0 torchaudio==2.6.0 --index-url https://download.pytorch.org/whl/cu124
pip install -r requirements.txt

git clone https://github.com/NARUTO-2024/WavBench.git
cd WavBench

Dataset

The data used in this project is available at WavBench Dataset hosted on Hugging Face.

You can load the dataset directly using the Hugging Face datasets library:

from datasets import load_dataset

# Load the dataset directly from Hugging Face
ds = load_dataset("WavBench/WavBench")

Alternatively, you can download the dataset to your local directory and use it directly.

1. Colloquial Expression

This category is divided into Basic and Pro subsets. Each subset contains tasks across 7 diverse cognitive domains:

2. Acoustic Interaction

This category evaluates the model's paralinguistic capabilities across three dimensions: Explicit Understanding, Explicit Generation, and Implicit.

Evaluation

Step 1: Run Inference

main.py is the unified entry point for all dataset types.

# Colloquial Inference (Basic) - With audio output
python main.py --model step_audio2 --data basic_code --audio_output

# Colloquial Inference (Pro) - With audio output
python main.py --model step_audio2 --data pro_math --audio_output

# Acoustic Single-turn Inference (With audio output)
python main.py --model step_audio2 --data acoustic_explicit_generation_emotion --audio_output

# Acoustic Multi-round Inference (With audio output)
python main.py --model step_audio2 --data acoustic_multi_round_generation --audio_output

# [Optional] Run with custom data directory
python main.py --model step_audio2 --data basic_code --data_dir /path/to/your/wavbench

Supported Arguments:

• --model: Model name (e.g., step_audio2).
• --data: Dataset name (e.g., basic_code, pro_math, acoustic_explicit_generation_emotion).
• --data_dir: Optional. Base directory for WavBench data (Default: ./wavbench). Use this argument if you have downloaded the dataset to a specific location other than the default.
• --audio_output: Important Flag. If set, the model generates audio files in addition to text.
  • Required for all Acoustic tasks (as evaluation relies on audio).
  • Optional for Colloquial tasks (useful if you want to check the TTS quality manually).

Step 2: Automatic Evaluation

evaluate.py uses LLMs (Gemini) to judge the responses based on the specific criteria of each subset.

# Option 1: Set API key via environment variable
export GOOGLE_API_KEY="your-api-key"

# Evaluate ALL Colloquial datasets
python evaluate.py --eval_type colloquial --dataset all

# Evaluate a SPECIFIC Colloquial dataset
python evaluate.py --eval_type colloquial --dataset basic_code

# Evaluate ALL Acoustic datasets
python evaluate.py --eval_type acoustic --dataset all

# Evaluate a SPECIFIC Acoustic dataset
python evaluate.py --eval_type acoustic --dataset explicit_generation_emotion

Supported Arguments:

• --eval_type: Choose between colloquial or acoustic.
• --dataset: Specific dataset name (e.g., basic_code) or use all to run the entire suite.

👇 Available Dataset Options (for --data / --dataset)

Category	Available Values
Basic	basic_code, basic_creative, basic_instruction, basic_logic, basic_math, basic_qa, basic_satety
Pro	pro_code, pro_creative, pro_instruction, pro_logic, pro_math, pro_qa, pro_satety
Explicit Generation	acoustic_explicit_generation_accent, acoustic_explicit_generation_age, acoustic_explicit_generation_audio, acoustic_explicit_generation_emotion, acoustic_explicit_generation_gender, acoustic_explicit_generation_lang, acoustic_explicit_generation_music, acoustic_explicit_generation_pitch, acoustic_explicit_generation_speed, acoustic_explicit_generation_volume
Explicit Understanding	acoustic_explicit_understanding_accent, acoustic_explicit_understanding_age, acoustic_explicit_understanding_audio, acoustic_explicit_understanding_emotion, acoustic_explicit_understanding_gender, acoustic_explicit_understanding_lang, acoustic_explicit_understanding_music, acoustic_explicit_understanding_pitch, acoustic_explicit_understanding_speed, acoustic_explicit_understanding_volume
Implicit	acoustic_implicit_age_generation, acoustic_implicit_emotion_generation, acoustic_implicit_pitch_generation, acoustic_implicit_speed_generation, acoustic_implicit_understanding
Multi-round	acoustic_multi_round_generation, acoustic_multi_round_understanding

Step 3: Get Statistics

statistics.py aggregates the evaluation results into a final report.

# Basic usage: Output to TXT file
python statistics.py --eval_dir ./eval_results --output ./statistics.txt

# Advanced usage: Output to TXT and CSV format simultaneously
python statistics.py --eval_dir ./eval_results --output ./statistics.txt --csv

Citation

If you use WavBench in your research, please cite the following paper:

@misc{li2026wavbenchbenchmarkingreasoningcolloquialism,
      title={WavBench: Benchmarking Reasoning, Colloquialism, and Paralinguistics for End-to-End Spoken Dialogue Models}, 
      author={Yangzhuo Li and Shengpeng Ji and Yifu Chen and Tianle Liang and Haorong Ying and Yule Wang and Junbo Li and Jun Fang and Zhou Zhao},
      year={2026},
      eprint={2602.12135},
      archivePrefix={arXiv},
      primaryClass={cs.CL},
      url={https://arxiv.org/abs/2602.12135}, 
}