VoxEval

🎉🎉🎉VoxEval has been accepted by ACL 2025 main conference!

Github repository for paper: VoxEval: Benchmarking the Knowledge Understanding Capabilities of End-to-End Spoken Language Models

Also check out our survey paper at Recent Advances in Speech Language Models: A Survey!

News

• 2025/06/16: We release the metric calculation code of VoxEval benchmark.
• 2025/03/23: We release the evaluation code of VoxEval benchmark.
• 2025/02/26: We release the Chain-of-Thought few-shot prompts for three mathematical reasoning subjects (Elementary Math, High School Math, and College Math) in VoxEval.

Info

VoxEval is a novel speech question-answering benchmark specifically designed to assess SLMs' knowledge understanding through purely speech-based interactions.

Below are the three highlights of our VoxEval benchmark:

• End-to-end speech-based evaluation: Both input and output are audio-based.
• Diverse audio conditions: VoxEval includes audio files featuring a variety of speakers, speaking styles, and audio qualities.
• Complex spoken evaluations: It supports advanced spoken assessments, including spoken math tasks.

Download Data

You can access our VoxEval Dataset Repository on 🤗 Hugging Face to directly download the dataset.

Below is the layout of the dataset folder. all_fewshot_examples folder contains the few shot examples for the evaluation. math_CoT_fewshot folder contains the few shot examples for evaluating the math subjects via Chain-of-Thought prompting. test folder contains the actual test data in VoxEval.

  # ├── Root Folder
  # │   ├── all_fewshot_examples
  # │       ├── alloy (different speaker voice)
  # │           ├── abstract_algebra_4o (different subjects)
  # │               ├── XXX.mp3
  # │               ├── ...
  # │           ├── ...
  # │       ├── echo
  # │       ├── ...
  # │   ├── test
  # │       ├── alloy (different settings)
  # │           ├── abstract_algebra_4o (different subjects)
  # │               ├── XXX.mp3
  # │               ├── ...
  # │           ├── ...
  # │       ├── echo
  # │       ├── ...

Evaluation Pipeline

The VoxEval_evaluation.py script contains the evaluation framework for VoxEval. It allows you to integrate and assess any SLMs within it. To evaluate a specific SLM, follow these steps:

1. Clone the repository of the SLM you wish to evaluate and download the model checkpoints provided by its authors.
2. Set up a conda environment as per the instructions given by the SLM's authors.
3. Create a file within the SLM's codebase that includes a function named e2e_evaluation(input_audio, sample_rate). This function should take audio samples and their sampling rate (loaded using torchaudio) as input. The input audio acts as an audio prompt, and the function should return the SLM's generated response audio.
4. Begin the evaluation process using the command provided below.

python VoxEval_evaluation.py \
    --source_path /path/to/VoxEval data/test \
    --fewshot_path /path/to/VoxEval data/all_fewshot_examples \
    --whisper_model_path /path/to/whisper-large-v3 \
    --target_path /path/to/save/the/evaluation/results \
    --eval_slm_path /path/to/target SLM codebase \
    --e2e_eval_file file_name_containing_the_e2e_evaluation_function \
    --timbre timbre \
    --shots num_shots \
    --prompt_mode regular_or_CoT \
    --cut_audio False

5. Run python metric_calculation.py to get the final accuracy of the evaluation.

GLM-4-Voice Evaluation (Example)

We give an example to evaluate GLM-4-Voice using VoxEval

1. Copy ./examples/VoxEval_eval_glm.py into the original repository of GLM-4-Voice.
2. Set up the GLM conda environment and activate it.
3. Start the evaluation using the command below (change the paths to your own):

python VoxEval_evaluation.py \
    --source_path /nfsdata/cuiwenqian/VoxEval/test \
    --fewshot_path /nfsdata/cuiwenqian/VoxEval/all_fewshot_examples \
    --whisper_model_path /nfsdata/cuiwenqian/hf_model_ckpt/whisper-large-v3 \
    --target_path /nfsdata/cuiwenqian/GLM-4-Voice/VoxEval_evaluation \
    --eval_slm_path /nfsdata/cuiwenqian/GLM-4-Voice \
    --e2e_eval_file VoxEval_eval_glm \
    --timbre alloy \
    --shots 5 \
    --prompt_mode regular \
    --cut_audio False

Evaluation results of existing end-to-end Spoken Language Models

License

The dataset is licensed under the Creative Commons Attribution 4.0.

Citation

@article{cui2025voxeval,
  title={VoxEval: Benchmarking the Knowledge Understanding Capabilities of End-to-End Spoken Language Models},
  author={Cui, Wenqian and Jiao, Xiaoqi and Meng, Ziqiao and King, Irwin},
  journal={arXiv preprint arXiv:2501.04962},
  year={2025}
}