Universal Audio Annotation Pipeline

Produces structured JSON annotations from any audio file, covering speech transcription, speaker diarization, emotions, vocal bursts, sound effects, and music.

Best configuration: Gemma-12B + DiCoW — Nemotron 3.5 words + VibeVoice/Sortformer diarization + DiCoW overlap-aware ASR, fused by a text-only Gemma-4-12B LLM (no audio in the final step). It is the highest-Reward pipeline on SoundScape-Bench (0.253) — rank 3 of all systems, nearly matching Gemini 3.5 Flash (0.256) and ahead of every other pipeline. (It trades precision for that recall: see the tradeoff note.)

⭐ Recommended: the default pipeline

A turnkey, end-to-end implementation lives in default_pipeline/ — the default, suggested configuration. It uses Nemotron 3.5 for the words, VibeVoice + Sortformer for diarization/timing, pyannote + DiCoW to transcribe overlapping speakers separately, the Whisper experts and the SFX LoRA sound-event detector — all fused by a text-only Gemma-4-12B model (it reads only the experts' outputs, not the audio). It produces expressive emotion & speaking-style captions, detailed sound-effect captions and a dedicated music segment type, strict diarization-only speaker identity, overlapping-speech segments, and explicit singing detection, plus per-clip JSON and a self-contained HTML report. The legacy audio MOSS-Audio-8B annotator (higher precision/F1) remains available via --fusion moss, as does the triple-ASR ensemble. See docs/default_pipeline.md for models, setup and run instructions.

cd default_pipeline && bash setup_environments.sh ./envs
export HF_TOKEN=...        # token with gated pyannote + SFX-LoRA access accepted
bash run_all.sh --audio /path/to/clips --workdir ./uaap_work --envs ./envs      # --fusion gemma (default)

⚡ With ≥2 GPUs the heavy stages (Gemma fusion, SFX, ASR…) are auto-sharded across GPUs over disjoint clips (≈N× faster, identical output; --gpus 0,1). Optional quality-neutral lower-VRAM fuser: export GEMMA_FILE=gemma-4-12b-it-UD-Q6_K_XL.gguf. See efficiency notes.

🎧 Live demo (20 samples — audio + predictions vs ground truth): demo · 📊 Full model comparison: comparison

🔗 Links

• Live predictions (example report): https://laion-ai.github.io/univeral-audio-annotation-pipeline/predictions/
• Model mirror on Hugging Face (all weights + code, self-contained): https://huggingface.co/laion/universal-audio-annotation-pipeline

Pipeline Architecture

The default configuration combines Nemotron 3.5 (words), VibeVoice + Sortformer (diarization/timing), pyannote + DiCoW (overlap-aware per-speaker ASR), the Whisper voice experts and the specialist sound-event pre-passes — all fused by a text-only Gemma-4-12B stage that reads the experts' outputs (not the audio):

┌───────────────────────────────────────────────────────────────────────┐
│                    INPUT: Audio File (any length)                      │
└───────────────────────────────────┬───────────────────────────────────┘
        ┌──────────────┬─────────────┴───────────┬───────────────────┐
        ▼              ▼                          ▼                   ▼
  ┌──────────┐  ┌────────────────┐     ┌──────────────┐   ┌────────────────────┐
  │ VibeVoice│  │ Nemotron 3.5 + │     │ pyannote     │   │ DiCoW              │
  │ ASR      │  │ Sortformer     │     │ diarization  │──▶│ diarization-cond.  │
  │ (diar/   │  │ (words)        │     │ + overlap    │   │ Whisper (per-spk,  │
  │  timing) │  │                │     │ detection    │   │ overlap-aware ASR) │
  └────┬─────┘  └───────┬────────┘     └──────┬───────┘   └─────────┬──────────┘
       │ timing/spk     │ words               │ overlaps            │ overlapped speech
       └────────┬───────┴─────────────┬───────┴──────────┬─────────┘
            ▼                      ▼                  ▼
  ┌──────────────────────┐  ┌──────────────────────────────┐
  │ Whisper experts (x3) │  │ Specialist sound-event prepass│
  │ emotion · timbre ·   │  │ • SFX LoRA (MOSS-8B-Instruct  │
  │ speaking-style        │  │   + laion sfx-lora r=128)     │
  │ (per utterance)      │  │ • Vocal-burst locator @0.88   │
  └──────────┬───────────┘  └───────────────┬──────────────┘
             └───────────────┬──────────────┘
                             ▼
        ┌──────────────────────────────────────────────┐
        │     Gemma-4-12B — TEXT-ONLY fusion (no audio) │
        │  fuses all expert text → final annotation     │
        │  Nemotron words on VibeVoice timeline         │
        │  DiCoW recovers overlapping/simultaneous spk  │
        │  detailed sound-event + dedicated music caps  │
        │  (legacy: MOSS-Audio-8B, audio, --fusion moss)│
        └───────────────────────┬──────────────────────┘
                                │  + deterministic gap-fill
                                ▼
        ┌──────────────────────────────────────────────┐
        │   OUTPUT: Structured JSON (covers full clip)  │
        │   [speech · vocal_burst · sound_event · music]│
        └──────────────────────────────────────────────┘

Output Format

The pipeline produces a JSON array of segment dictionaries. Four segment types:

Speech segment

{
  "type": "speech",
  "start_time": 2.31,
  "end_time": 5.87,
  "transcription": "I can't believe you actually did that",
  "speaker_id": "speaker_1",
  "emotion": "anger_3",
  "age": "adult_30s",
  "gender": "female",
  "voice_timbre": "alto, warm, slightly raspy",
  "speaking_style": "confrontational, accusatory, raised voice",
  "language": "en",
  "accent": "American Midwest",
  "speaking_rate": "fast"
}

Vocal burst segment

{
  "type": "vocal_burst",
  "start_time": 5.87,
  "end_time": 6.14,
  "speaker_id": "speaker_1",
  "vocal_burst": "scoff",
  "emotion": "contempt_2"
}

Sound event segment

{
  "type": "sound_event",
  "start_time": 3.10,
  "end_time": 5.20,
  "description": "Medium-sized dog barking aggressively in the background",
  "loudness": "moderate"
}

Music segment

{
  "type": "music",
  "start_time": 9.80,
  "end_time": 15.10,
  "description": "Upbeat acoustic folk-pop: brightly strummed steel-string guitar and light tambourine, mid-tempo ~110 BPM, major key, warm and nostalgic mood, no vocals",
  "loudness": "moderate"
}

Field Reference

Quick Start

Installation

pip install -r requirements.txt

Run the pipeline

# DEFAULT pipeline (Gemma-12B + DiCoW, text-only fusion): use the staged runner
cd default_pipeline
bash run_all.sh --audio input.wav --workdir ./uaap_work --envs ./envs       # --fusion gemma (default)

The default Gemma-12B + DiCoW configuration runs as the staged default_pipeline/ (each stage in its own venv — llama.cpp, pyannote, DiCoW and the ASR toolkits pin incompatible deps, so they can't share one process). The single-process pipeline/run_pipeline.py below is a legacy reference for the MOSS-Audio configurations only.

Configurations

⭐ Gemma-12B + DiCoW is the default (bash run_all.sh ... --fusion gemma). It adds pyannote diarization + DiCoW overlap-aware ASR, then fuses every expert's text output with Gemma-4-12B (no audio in the final step) — plus the SFX LoRA detector, vocal-burst locator (0.7) + captioner, emotion/ style captions, dedicated music segments, overlapping-speech segments and full-timeline coverage. Use --fusion moss for the audio MOSS-Audio-8B annotator (higher precision). See docs/default_pipeline.md.

Model Requirements

Models load/unload sequentially (each stage in its own venv), so peak VRAM ≈ the largest single stage (~16–18 GB); the default runs comfortably on one 24 GB GPU, faster on two.

Gated models (request access, then export HF_TOKEN=...): pyannote/segmentation-3.0, pyannote/speaker-diarization-3.1, and laion/moss-audio-sfx-lora-v4 (or run --no-sfx). The Gemma GGUF downloads automatically. Full model table with links: docs/default_pipeline.md.

Evaluation Results

SoundScape-Bench (200 multilingual soundscapes, automatic answer-key scoring)

The current default was selected on SoundScape-Bench — 200 held-out soundscapes (EN/ZH/FR/DE/ES/NL, ~25 % overlapping speech) built from understood pieces so every event has an exact answer key. The headline Reward = IoU(timing) × content, where content is a weighted mix of caption cosine and (1 − WER) for speech, averaged over all answer-key events.

The default Gemma-12B + DiCoW is the highest-Reward pipeline of all (rank 3 overall, nearly matching Gemini 3.5 Flash and well above the audio MOSS configs). A text-only LLM that faithfully fuses strong experts wins on timing (IoU) and transcription (WER), and DiCoW recovers overlapping/simultaneous speakers (cleanly separating e.g. an English and a Mandarin speaker talking at once).

⚖️ Precision tradeoff (read before deploying): because the Gemma fuser has no audio, it cannot reject candidate events it can't hear, so it over-generates — hallucination ~43 % and F1 ~0.15, versus the audio MOSS-Audio-8B config's 27 % / 0.191. If you need maximum precision / fewest spurious events, run --fusion moss. The default optimizes the recall-oriented Reward at lower cost (a ~12B text model, no audio in the final step).

📊 Interactive comparison: soundscape_comparison.html · 🎧 20-sample audio demo (predictions vs ground truth): gemma12_dicow_demo.html

Vocal-burst detection threshold (how we chose 0.88)

The vocal-burst pre-pass is a two-model ensemble: the detector laion/vocalburst-locator finds where bursts occur, and the captioner laion/vocalburst-captioning-whisper (a Whisper-small fine-tune of laion/sound-effect-captioning-whisper on laion/improved_synthetic_vocal_burts; val clap_sim 0.251 vs 0.190 untuned) describes what each one is.

To pick the detector's confidence threshold we swept 0.85 → 0.92 (1% steps) over 150 audio clips (merge_gap=0.3 s, min_dur=0.5 s). For every (clip × threshold) the detected segments were captioned, and the audio + (start, end, caption) list was sent to Gemini 3.1 Pro, which rated three axes 0–5 (5 = perfect): caption quality, timestamp accuracy, and completeness (cover ALL real bursts, penalizing misses and false positives) — 1,200 independent judgments. overall = mean of the three.

Scores cluster tightly across the band; threshold 0.88 is the best overall and is now the pipeline default (UAAP_VB_THRESHOLD, VocalBurstLocator.detect(threshold=0.88)). Timestamp accuracy is consistently strong (~4.0); completeness is the weakest axis and drops at 0.91–0.92 as real bursts get missed. Full interactive report (audio + predictions + per-clip Gemini scores for the top-3 thresholds): on the locator model card.

Legacy LLM-judged eval (60 scenes, Gemini 3.1 Pro, 0–5 scale)

Earlier configuration sweep that selected the previous triple-ASR default:

Combined = equal-weight average of Synthetic and YouTube scores.

For detailed evaluation results, see docs/evaluation_results.md.

Interactive evaluation grid: GitHub Pages

Repository Structure

default_pipeline/        # ⭐ Default recommended configuration (turnkey scripts)
  setup_environments.sh  # Build the per-component virtual-envs
  run_all.sh             # Run all stages end-to-end
  prepare_audio.py       # Stage 0: decode + index
  workers/               # One script per stage:
    stage1a_vibevoice.py #   VibeVoice-ASR (diarization / timing authority)
    stage1_nemotron_sortformer.py #  Nemotron 3.5 + Sortformer (default word source)
    stage_pyannote_diar.py #  ⭐ pyannote diarization + overlap detection (default)
    stage_dicow.py       #   ⭐ DiCoW diarization-conditioned overlap ASR (default)
    stage5_gemma_fusion.py #  ⭐ Gemma-12B TEXT-only fusion — DEFAULT final stage
    stage1b_parakeet.py  #   Parakeet TDT v3 + Sortformer (legacy ensemble option)
    stage1c_qwen3.py     #   Qwen3-ASR + ForcedAligner (legacy ensemble option)
    stage2_whisper_experts.py  # emotion/timbre/style
    stage3_sfx_lora.py   #   SFX LoRA sound events
    stage3b_vocalburst.py#   Vocal-burst locator + captioner
    stage4_moss_annotator.py   # MOSS-Audio final annotation (legacy, --fusion moss)
  build_report.py        # Self-contained HTML report

pipeline/
  run_pipeline.py        # Single-process reference runner (MOSS configs only)
  asr_vibevoice.py       # VibeVoice-ASR component
  asr_nemotron.py        # Nemotron 3.5 + Sortformer (default word source)
  diarize_pyannote.py    # ⭐ pyannote diarization + overlap detection
  asr_dicow.py           # ⭐ DiCoW diarization-conditioned overlap ASR
  gemma_fusion.py        # ⭐ Gemma text-only LLM fusion (default final stage)
  asr_parakeet.py        # Parakeet TDT v3 + Sortformer (legacy ensemble option)
  asr_qwen3.py           # Qwen3-ASR-1.7B + ForcedAligner (legacy ensemble option)
  whisper_experts.py     # Emotion/timbre/style Whisper models
  sfx_lora.py            # LoRA SFX sound event detection
  vocalburst_locator.py  # Vocal-burst locator + sound-effect captioner
  moss_annotator.py      # MOSS-Audio-8B-Thinking annotation (legacy final stage)
  utils.py               # Shared utilities (incl. full-timeline gap-fill)

evaluation/
  eval_triple_asr.py     # Full evaluation script
  gemini_judge.py        # Gemini evaluation scorer
  build_html_report.py   # HTML report generator

docs/
  default_pipeline.md    # ⭐ Default configuration: models, links, setup, run guide
  pipeline_details.md    # Per-component details
  evaluation_results.md  # Full evaluation results
  training_lora.md       # LoRA training details
  eval_grid/index.html   # Interactive evaluation grid

examples/
  sample_output.json     # Example pipeline output
  sample_predictions/    # Sample prediction JSONs

Links

ASR

• VibeVoice-ASR: microsoft/VibeVoice-ASR · code
• Parakeet TDT v3: nvidia/parakeet-tdt-0.6b-v3 · Sortformer: nvidia/diar_sortformer_4spk-v1
• Qwen3-ASR: Qwen/Qwen3-ASR-1.7B · Qwen/Qwen3-ForcedAligner-0.6B

Voice experts

• Emotion: laion/BUD-E-Whisper · Timbre: laion/timbre-whisper · Style: laion/voice-tagging-whisper

Sound events

• SFX LoRA: laion/moss-audio-sfx-lora-v4 (gated) on OpenMOSS-Team/MOSS-Audio-8B-Instruct
• Vocal-burst locator: laion/vocalburst-locator
• Vocal-burst captioner: laion/vocalburst-captioning-whisper (fine-tune of laion/sound-effect-captioning-whisper)

Annotator & MOSS-Audio

• OpenMOSS-Team/MOSS-Audio-8B-Thinking · source code: github.com/OpenMOSS/MOSS-Audio

• Evaluation Grid: GitHub Pages

License

Apache 2.0