SoundScape-Bench

200 held-out multilingual soundscapes with exact, automatically-gradable answer keys for evaluating "universal audio annotation" — the task of describing everything audible in a clip (speech, who/when/ what/which-language/how-it-is-said, sound effects, music, and vocal bursts) as one structured JSON list.

It is the benchmark for the LAION Universal Audio Annotation Pipeline (UAAP).

Why it exists

Every clip is built by gluing together pieces we already understand — a known speech utterance, a known sound effect, known music, a known vocal burst — placed on a timeline. Because we assembled the clip, we know the perfect answer by construction, so grading is 100% automatic (no human raters, no LLM judge needed for the ground truth).

Contents

Answer-key format (UAAP schema)

Each clips/clip_NNN.json is {"clip_id","duration","num_speakers","languages":[...],"events":[...]}. Each event is one of four types (times in seconds; events may overlap):

{"type":"speech","start_time":2.31,"end_time":5.87,"transcription":"...","speaker_id":"speaker_1",
 "language":"en","emotion":"clearly intense anger laced with disappointment","age":"adult_30s",
 "gender":"female","voice_timbre":"alto, warm, slightly raspy","speaking_style":"confrontational bark"}
{"type":"vocal_burst","start_time":5.87,"end_time":6.14,"speaker_id":"speaker_1","vocal_burst":"scoff",
 "emotion":"slight contempt","transcription":null}
{"type":"sound_event","start_time":3.10,"end_time":5.20,"description":"Medium dog barking","loudness":"moderate"}
{"type":"music","start_time":9.80,"end_time":15.10,"description":"Upbeat folk-pop, strummed guitar, ~110 BPM","loudness":"moderate"}

answers.jsonl has one such object per line, plus an audio field pointing at the mp3.

How it is scored (the "Reward" metric)

Per-type Hungarian matching between predicted and answer-key events; each matched pair scores IoU(timing) × content, where content for speech = ½·cos(emotion/style caption) + ½·(1 − WER on the transcription), and for sound/music/burst = caption cosine similarity (via google/embeddinggemma-300m); WER via jiwer (per-character for Chinese). The headline Reward is the average over all answer-key events (a missed event scores 0). The scoring harness and metric definition live in the pipeline repo.

Leaderboard (Reward, full 200)

Interactive comparison: soundscape_comparison.html.

Sources & licensing

Clips are mixed from: speech — laion/emolia-hq (EN/DE/FR), facebook/multilingual_librispeech (ES/NL), amphion/Emilia-Dataset (ZH); speech "how-it's-said" captions re-generated with laion/BUD-E-Whisper_V1.21; sound effects — mitermix/audioset-with-grounded-captions; music — laion/captioned-ai-music-snippets; vocal bursts — laion/improved_synthetic_vocal_burts. Each underlying source keeps its own license; the SFX layer is AudioSet-derived (YouTube-sourced), so this benchmark is intended for research / evaluation use. Released CC-BY-4.0 for the packaging and answer keys; please attribute LAION and the upstream datasets.

Usage

from datasets import load_dataset
ds = load_dataset("laion/soundscape-bench", split="test")   # reads answers.jsonl
ex = ds[0]; print(ex["clip_id"], ex["duration"], ex["languages"], len(ex["events"]))
# audio file: ex["audio"]  ->  clips/clip_000.mp3  (download the clips/ folder alongside)