VoiceBench: Benchmarking LLM-Based Voice Assistants

🏆 Leaderboard | 📄 Paper | 🤗 Data

We encourage new result submissions through the issue tracker. The leaderboard will be updated accordingly.

News

• 2026.04.20 Check out HalluAudio, a comprehensive benchmark for hallucination detection in LALMs.
• 2025.04.20 Released wildvoice, a crowd-sourced dataset comprising human-recorded speech with diverse accents.
• 2025.04.12 Released bbh, a crowd-sourced dataset comprising human-recorded speech, for evaluating the reasoning ability of voice assistants.
• 2024.12.11 Updated the VoiceBench Leaderboard to include mmsu.
• 2024.12.10 Added a curated list of awesome voice assistants.
• 2024.11.24 Expanded the test samples in VoiceBench to include mmsu, covering 12 diverse domains from mmlu-pro.
• 2024.11.12 Updated the VoiceBench Leaderboard to include: 1) Mini-Omni2, GPT-4o-Audio, and Whisper-v3+GPT-4o, and 2) multiple-choice QA from OpenBookQA.
• 2024.10.30 Expanded the test samples in VoiceBench to include: 1) the complete set of open-ended QA from alpacaeval, and 2) multiple-choice QA from openbookqa.

Table of Contents

• Setup
• Dataset
• Evaluation
• Awesome Voice Assistants
• Citation

Setup

conda create -n voicebench python=3.10
conda activate voicebench
pip install torch==2.1.2 torchvision==0.16.2 torchaudio==2.1.2 --index-url https://download.pytorch.org/whl/cu121
pip install xformers==0.0.23 --no-deps
pip install -r requirements.txt

Dataset

The data used in this project is available at VoiceBench Dataset hosted on Hugging Face.

You can access it directly via the link and integrate it into your project by using the Hugging Face datasets library.

How to Use the Dataset

To load the dataset in your Python environment:

from datasets import load_dataset

# Load the VoiceBench dataset
# Available subset: alpacaeval, commoneval, sd-qa, ifeval, advbench, ...
dataset = load_dataset("hlt-lab/voicebench", 'alpacaeval')

Available Data

PS: alpacaeval contains helpful_base and vicuna data, while alpacaeval_full is constructed with the complete data. alpacaeval_full is used in the leaderboard.

Evaluation

Step 1: Get the Voice Assistant's Response

To obtain the responses from the voice assistant model, run the following command:

python main.py --model naive --data alpacaeval --split test --modality audio

Supported Arguments:

• --model: Specifies the model to use for generating responses. Replace naive with the model you want to test (e.g., qwen2, diva).
• --data: Selects the subset of the dataset. Replace alpacaeval with other subsets like commoneval, sd-qa, etc., depending on your evaluation needs.
• --split: Chooses the data split to evaluate.
  • For most datasets (alpacaeval, commoneval, ifeval, advbench), use test as the value.
  • For the sd-qa subset, you should provide a region code instead of test, such as aus for Australia, usa for the United States, etc.
• --modality: Use audio for spoken instructions, text for text-based instructions.

This will generate the output and save it to a file named naive-alpacaeval-test-audio.jsonl.

Step2: Automatic GPT-4 Evaluation

For datasets alpacaeval, commoneval, wildvoice, and sd-qa, we use gpt-4o-mini to evaluate the responses. Run the following command to get the GPT score:

python api_judge.py --src_file naive-alpacaeval-test-audio.jsonl

The GPT evaluation scores will be saved to result-naive-alpacaeval-test-audio.jsonl.

Note: This step should be skipped for other datasets, as they are not evaluated using GPT-4.

Step3: Get the Final Results

To generate the final evaluation results, run:

python evaluate.py --src_file result-naive-alpacaeval-test-audio.jsonl --evaluator open

Supported Arguments:

• --evaluator: Specifies the evaluator type:
  • Use open for alpacaeval, commoneval, and wildvoice.
  • Use qa for sd-qa.
  • Use ifeval for ifeval.
  • Use harm for advbench.
  • Use mcq for openbookqa and mmsu.
  • Use bbh for bbh.

Awesome Voice Assistants

Citation

If you use the VoiceBench in your research, please cite the following paper:

@article{chen2024voicebench,
  title={VoiceBench: Benchmarking LLM-Based Voice Assistants},
  author={Chen, Yiming and Yue, Xianghu and Zhang, Chen and Gao, Xiaoxue and Tan, Robby T. and Li, Haizhou},
  journal={arXiv preprint arXiv:2410.17196},
  year={2024}
}