Do Audio LLMs Listen or Read? Analyzing and Mitigating Paralinguistic Failures with VoxParadox
4
18 commits
1 linked in READMEs
updated Jun 3, 2026
Do Audio LLMs Listen or Read? Analyzing and Mitigating Paralinguistic Failures with VoxParadox
Jiacheng Pang*, Ashutosh Chaubey*, Mohammad Soleymani
University of Southern California
* Equal contribution
An adversarial speech QA benchmark for paralinguistic understanding in Audio LLMs. Each example is built around a controlled linguistic–acoustic contradiction: the transcript explicitly asserts an incorrect paralinguistic attribute, while the audio reliably conveys the correct one. Models that defer to transcript content are misled; models that listen are not.
2,000 MCQs across 10 paralinguistic tasks (200 each).
from datasets import load_dataset
ds = load_dataset("IHP-Lab/VoxParadox", split="test")
print(ds[0]) # includes `audio` (decoded), `question`, `choice_a..d`,
# `answer_gt`, `adversarial_labels`, `task_name`, `id`
y_true (audio) and y_adv (transcript) are disjoint by construction.
Task (task_name) | Acoustic attribute the model must recover |
|---|---|
age_prediction | Speaker's age group |
gender_prediction | Speaker's gender |
emotion_recognition | Emotion conveyed by delivery (high-contrast pairs) |
intonation_perception | Rising vs. falling intonation |
speaker_identity_recognition | Which segment shares a speaker with a queried segment |
total_speaker_counting | Number of distinct speakers |
pitch_comparison | Ordering of three segments by pitch |
volume_comparison | Ordering of three segments by loudness |
speed_comparison | Ordering of three segments by speaking rate |
vocal_range_comparison | Ordering of three segments by pitch range |
.
├── metadata.jsonl # one record per example (loaded by `datasets`)
├── voxparadox.json # same content as JSON array (for direct inspection)
├── audio/ # 2,000 wav files
├── eval.py # evaluation script (GT accuracy + ALA)
├── assets/ # logo
└── LICENSE # USC Research License
| Field | Type | Description |
|---|---|---|
id | string | {task_name}__{N}, with N running 0–199 within each task. |
task_name | string | One of the 10 tasks above. |
file_name / audio_path | string | Path to the audio clip, relative to this directory. |
question | string | The MCQ question prompt. |
choice_a / choice_b / choice_c / choice_d | string | The four answer options. |
answer_gt | string | The acoustic ground-truth y_true (one of the four choices). |
adversarial_labels | list[string] | The transcript-implied label(s) y_adv. Single-element for most tasks; 2 elements for the four *_comparison tasks. |
Two complementary metrics:
answer_gt. Higher is better; reflects use of acoustic evidence.adversarial_labels. Higher ALA means more transcript-following.Run eval.py on a JSONL of model predictions (one record per line, fields id and response):
python eval.py --predictions preds.jsonl
Example prediction record:
{"id": "age_prediction__0", "response": "(C) Elderly adult."}
The script parses A/B/C/D from the response (letter-first, then choice-text
fallback), prints per-task and overall GT/ALA, and optionally writes a JSON
report with --report report.json.
Released under the USC Research License (research and non-profit use only;
commercial use requires a separate license). See LICENSE for the
full text.
Audio was synthesized via commercial TTS engines (ElevenLabs, GPT-4o, Microsoft Azure); any commercial reuse of the audio is additionally subject to those vendors' terms of service.
@inproceedings{pang2026voxparadox,
title = {Do Audio LLMs Listen or Read? Analyzing and Mitigating Paralinguistic Failures with VoxParadox},
author = {Pang, Jiacheng and Chaubey, Ashutosh and Soleymani, Mohammad},
booktitle = {Proceedings of the International Conference on Machine Learning (ICML)},
year = {2026}
}
Do Audio LLMs Listen or Read? Analyzing and Mitigating Paralinguistic Failures with VoxParadox
4
18 commits
1 linked in READMEs
updated Jun 3, 2026
Do Audio LLMs Listen or Read? Analyzing and Mitigating Paralinguistic Failures with VoxParadox
Jiacheng Pang*, Ashutosh Chaubey*, Mohammad Soleymani
University of Southern California
* Equal contribution
An adversarial speech QA benchmark for paralinguistic understanding in Audio LLMs. Each example is built around a controlled linguistic–acoustic contradiction: the transcript explicitly asserts an incorrect paralinguistic attribute, while the audio reliably conveys the correct one. Models that defer to transcript content are misled; models that listen are not.
2,000 MCQs across 10 paralinguistic tasks (200 each).
from datasets import load_dataset
ds = load_dataset("IHP-Lab/VoxParadox", split="test")
print(ds[0]) # includes `audio` (decoded), `question`, `choice_a..d`,
# `answer_gt`, `adversarial_labels`, `task_name`, `id`
y_true (audio) and y_adv (transcript) are disjoint by construction.
Task (task_name) | Acoustic attribute the model must recover |
|---|---|
age_prediction | Speaker's age group |
gender_prediction | Speaker's gender |
emotion_recognition | Emotion conveyed by delivery (high-contrast pairs) |
intonation_perception | Rising vs. falling intonation |
speaker_identity_recognition | Which segment shares a speaker with a queried segment |
total_speaker_counting | Number of distinct speakers |
pitch_comparison | Ordering of three segments by pitch |
volume_comparison | Ordering of three segments by loudness |
speed_comparison | Ordering of three segments by speaking rate |
vocal_range_comparison | Ordering of three segments by pitch range |
.
├── metadata.jsonl # one record per example (loaded by `datasets`)
├── voxparadox.json # same content as JSON array (for direct inspection)
├── audio/ # 2,000 wav files
├── eval.py # evaluation script (GT accuracy + ALA)
├── assets/ # logo
└── LICENSE # USC Research License
| Field | Type | Description |
|---|---|---|
id | string | {task_name}__{N}, with N running 0–199 within each task. |
task_name | string | One of the 10 tasks above. |
file_name / audio_path | string | Path to the audio clip, relative to this directory. |
question | string | The MCQ question prompt. |
choice_a / choice_b / choice_c / choice_d | string | The four answer options. |
answer_gt | string | The acoustic ground-truth y_true (one of the four choices). |
adversarial_labels | list[string] | The transcript-implied label(s) y_adv. Single-element for most tasks; 2 elements for the four *_comparison tasks. |
Two complementary metrics:
answer_gt. Higher is better; reflects use of acoustic evidence.adversarial_labels. Higher ALA means more transcript-following.Run eval.py on a JSONL of model predictions (one record per line, fields id and response):
python eval.py --predictions preds.jsonl
Example prediction record:
{"id": "age_prediction__0", "response": "(C) Elderly adult."}
The script parses A/B/C/D from the response (letter-first, then choice-text
fallback), prints per-task and overall GT/ALA, and optionally writes a JSON
report with --report report.json.
Released under the USC Research License (research and non-profit use only;
commercial use requires a separate license). See LICENSE for the
full text.
Audio was synthesized via commercial TTS engines (ElevenLabs, GPT-4o, Microsoft Azure); any commercial reuse of the audio is additionally subject to those vendors' terms of service.
@inproceedings{pang2026voxparadox,
title = {Do Audio LLMs Listen or Read? Analyzing and Mitigating Paralinguistic Failures with VoxParadox},
author = {Pang, Jiacheng and Chaubey, Ashutosh and Soleymani, Mohammad},
booktitle = {Proceedings of the International Conference on Machine Learning (ICML)},
year = {2026}
}