AcousticBench

AcousticBench is an audio-perception benchmark for Large Audio-Language Models (LALMs). This repository contains the evaluation harness, Dockerized model servers, and reference results for reproducing every number in the paper.

3,200 scored + 600 diagnostic = 3,800 questions total.

---

Required for all steps below

Two API keys are needed regardless of which models you evaluate:

Save them:

# OpenAI key — put in .env (auto-loaded by run.py)
cp example.env .env
# Edit .env and set OPENAI_API_KEY=sk-...

# HuggingFace token — save where Docker containers can find it
mkdir -p ~/.cache/huggingface
echo "hf_YOUR_TOKEN" > ~/.cache/huggingface/token

Additional keys (GEMINI_API_KEY) are only needed if you want to run Gemini models specifically.

---

Prerequisites

nvidia-smi
docker run --rm --gpus all nvidia/cuda:12.4.0-base-ubuntu22.04 nvidia-smi

API models (Gemini, GPT-4o, Cascade) need no GPU — just API keys and internet.

---

Setup

git clone <repo-url> && cd acousticbench-public
python3 -m venv .venv && source .venv/bin/activate
pip install -e ".[all]"

# Set up API keys (see "Required for all steps below" above)
cp example.env .env
# Edit .env and set OPENAI_API_KEY=sk-...

# Extract benchmark data (use the actual filename you received)
python3 -c "import zipfile; zipfile.ZipFile('YOUR_ARCHIVE.zip').extractall('archive')"

# Verify extraction
python run.py --verify-checksums --task all
# Expected: "All 3800 questions passed checksum verification."

Important: Run source .venv/bin/activate in every new terminal session before using run.py or any scripts.

---

Level 1 — Run the models yourself

Run any model on any task and get accuracy numbers.

API model

export GEMINI_API_KEY=your_key
python run.py --model Gemini-3.1-Pro --task rad --max-samples 3 --seed 42

Open-weight model on your GPU

# Start the model server (builds Docker image on first run, downloads weights from HuggingFace)
./scripts/docker_serve.sh voxtral_mini_3b

# Export the endpoint (printed by the script)
export ACOUSTICBENCH_VOXTRAL_MINI_3B_ENDPOINT=http://localhost:8009

# Run the evaluation (--max-samples 3 for a quick test; omit for the full task)
python run.py --model Voxtral-Mini-3B --task rad --max-samples 3 --seed 42

# Stop when done
./scripts/docker_serve.sh voxtral_mini_3b --stop

CLI flags

Where results go

results/<Model>-<task>-<backend>-seed<N>-<timestamp>/
  responses.jsonl    # one JSON line per question
  config.json        # run configuration
  metrics.json       # aggregate accuracy

---

Level 2 — Compare your results to our reference JSONLs

This verifies that your local GPU produces the same answers we reported. Add --compare-reference to any run.py command and it will automatically compare each run against the matching file in reference_trajectories/.

Quick check (5 minutes)

./scripts/docker_serve.sh voxtral_mini_3b
export ACOUSTICBENCH_VOXTRAL_MINI_3B_ENDPOINT=http://localhost:8009

python run.py --model Voxtral-Mini-3B --task rad --max-samples 10 --seed 42 --compare-reference
# Expected output: "Reference comparison: 10/10 (100.0%) match ..."

Expected match rates

Reference trajectories layout

reference_trajectories/
  <Model>/
    <task>/
      run_1.jsonl       # seed 42
      run_2.jsonl       # seed 43
      run_3.jsonl       # seed 44
      summary.json      # mean accuracy, std, per-run breakdown

---

Models

API models (no GPU needed)

Open-weight models (GPU + Docker)

Each model's endpoint env var is printed by docker_serve.sh when starting. See ./scripts/docker_serve.sh --list for the full list.

Kimi LoRA finetunes (paper §4.2)

Requires Kimi LoRA adapter weights extracted into ./archive/. Download from Kaggle:

pip install kagglehub
python3 -c "
import kagglehub, shutil
path = kagglehub.dataset_download('blueberrystonekitten/acousticbench-kimi-rap-finetunes')
shutil.copytree(path, 'archive', dirs_exist_ok=True)
print('Adapter weights extracted to ./archive/')
"

Important: LoRA reference trajectories were run with --split eval. Use --split eval when comparing against them. The LoRA Docker container needs ~60 GB of free disk on first boot to export the merged model (cached for subsequent runs).

---

JSONL schema

Each run_*.jsonl (and each responses.jsonl from your own runs) has one JSON line per question:

---

Handling refusals and errors

A small fraction of responses are abstentions or verbose text the judge can't parse. If you see null values in chosen:

# Retry refusals up to 3 times
python scripts/rerun_nones.py --model MODEL_NAME --concurrency 8 --max-retries 3 \
    results/MODEL_NAME-*-seed42-*/responses.jsonl

# Re-judge remaining unparseable responses via LLM
python scripts/rejudge.py results/MODEL_NAME-*/responses.jsonl

---

Notes

Greedy decoding. All open-weight models use temperature=0, do_sample=False. Run-to-run variance is zero or near-zero. Small differences across GPU types come from flash-attention floating-point nondeterminism, not sampling.

Paper configuration. All reference trajectories were produced on A100-80GB GPUs, with 3 runs (seeds 42, 43, 44) per (model, task).

Train/eval splits. Each task has train and eval halves. Main leaderboard: --split all. LoRA lift table (paper Table 2): --split eval.

MiniCPM-o-4.5 merges paired audio clips with a 1-second silence gap for 2-clip tasks because its chat interface accepts only single audio input.

LLM judge. The harness uses gpt-4o-mini to extract A/B labels from free-text responses. Requires OPENAI_API_KEY.

Zip format. The benchmark archive uses Zip64. System unzip on many Linux distributions can't handle this — use the Python zipfile command shown above.

First Docker run. Downloads model weights from HuggingFace (8-70 GB depending on model). Weights cache at ~/.cache/huggingface and reuse on subsequent runs.