DAVE: A Decoupled Audio-Visual Enhancement Framework for Real-World Speech Separation

Official implementation of DAVE, our system for the ISCSLP 2026 Real-World AVSE Challenge — audio-visual speech enhancement and target speaker extraction on real-world meeting recordings. DAVE decouples audio reconstruction from visual speaker attribution: an audio-only separation backbone (TIGER-M, 2.56M) reconstructs two anonymous speech streams; a four-cue weighted-fusion module assigns speaker identities from face video; a selective enhancement chain (scene routing → GAN denoising → loudness normalization) finalizes output.

• Paper: DAVE: A Decoupled Audio-Visual Enhancement Framework for Real-World Speech Separation — arXiv:2608.09288
• Model weights: bundled in this repository (model/real_avse_tiger_m.ckpt, 2.56M parameters, md5 15800af8) · mirrored on Hugging Face
• Training dataset: DAVE-Corpus — 12,000 clean-provenance mixture pairs (CC BY-SA 4.0)
• Challenge: ISCSLP 2026 Real-World AVSE Challenge

Highlights

• Robust to degraded video by design — visual cues only vote on identity assignment; the separation itself never consumes video, so corrupted visual input cannot corrupt the reconstructed audio.
• Tiny, self-contained backbone — 2.56M parameters, weights bundled in this repo; inference runs after pip install torch numpy soundfile, nothing else to download.
• Top-tier official results — Track 1: 4th/17 and Track 2: 3rd/13 on the official leaderboard (seven-metric rank-mean).
• 20 sample pairs with a listening page — index.html plays every mixture/separation pair in the browser (serve it with GitHub Pages or any static server; also mirrored on Hugging Face).

---

News

• [2026-08] Paper released: DAVE: A Decoupled Audio-Visual Enhancement Framework for Real-World Speech Separation (arXiv:2608.09288).
• [2026-08] Ranked Track 1: 4th / 17 and Track 2: 3rd / 13 in the ISCSLP 2026 Real-World AVSE Challenge (seven-column rank-mean).
• [2026-08] Code and model weights released.

🏆 Challenge Results (official leaderboard, seven-column rank-mean)

Track 1 — 4th of 17 teams (top 5 shown):

Track 2 — 3rd of 13 teams (top 5 shown):

Overview

Mixed audio is separated by the audio-only TIGER-M backbone into two anonymous streams. In parallel, four cues extracted against the face tracks — voiceprint (×3.0), StableSyncNet (×1.5), SyncNet (×1.0), lip keypoints (×0.7) — are normalized and fused into one identity vote per scene. A selective enhancement chain (scene routing → MossFormerGAN denoising → loudness normalization) finalizes the no-reference partition.

Repository Structure

DAVE/
├── inference.py                # Self-contained separation CLI (torch only) — runnable
├── index.html                  # Listening page for the 20 sample pairs (GitHub Pages ready)
├── run_train.sh                # Training launcher (final-stage configuration + curriculum notes)
├── src/
│   ├── models/                 # Bundled TIGER-M backbone + WeSpeaker ResNet34 extractor
│   ├── train.py                # Six-loss DDP trainer (model/PIT/speaker modules bundled)
│   └── synth/                  # Training-data pipeline (mixtures + transcripts)
│       ├── synthesize.py       #   RIR-based two-speaker mixture synthesis
│       ├── synthesize_misp.py  #   real-acoustic mixtures from far-field MISP segments
│       ├── synthesize_pure.py  #   direct-sum mixtures with exact-zero residual
│       └── transcribe.py       #   Fun-ASR-Nano transcript labeling (CER-loss targets)
├── reference/                  # Reference code (depends on challenge intermediates; not standalone)
│   ├── attribution/            #   four-cue weighted-fusion speaker attribution
│   ├── enhance/                #   scene routing + MossFormerGAN + loudness norm
│   └── evaluation/             #   seven-metric evaluation (evaluation-aligned engines)
├── model/real_avse_tiger_m.ckpt   # TIGER-M weights (2.56M)
├── samples/                    # 20 playable example pairs (mix/s1/s2, EchoSet, Apache-2.0)
├── assets/                     # Framework figure
├── configs/paths.example.yaml
├── manifests/                  # Synthesis recipe manifests (no copyrighted audio)
├── requirements.txt · DATA_LICENSE.md · LICENSE · NOTICE

Runnable out of the box: inference.py (+ src/models/). Runnable after preparing data and assets (synthesized corpus, WeSpeaker checkpoint, Fun-ASR teacher — see DATA_LICENSE.md): src/synth/ and src/train.py via run_train.sh; no external separation repository is required. Reference only (depend on challenge intermediates): everything under reference/.

Inference (Quick Start)

Separation is self-contained — the TIGER-M model is bundled under src/models/, so only torch, numpy, soundfile are needed (no external repo to clone):

pip install torch numpy soundfile

# Separate one mixture (or a directory of mixtures) into two streams:
python inference.py --mixture samples/echoset_01/mix.wav --out output
python inference.py --mixture path/to/dir --out output --device cuda

Output: output/<name>/s1.wav, s2.wav (16 kHz mono). The backbone is audio-only; speaker attribution (from video) and the enhancement chain are separate stages under reference/attribution/ and reference/enhance/.

Tested with Python 3.11 and PyTorch 2.x; inference runs on CPU or a single GPU.

Listening Samples

20 example pairs live under samples/. GitHub itself does not play audio inline, so a ready-made listening page is included: index.html plays every mixture/separation pair in the browser.

• Hosted: enable GitHub Pages (Settings → Pages → deploy from main, root) and the page is served at https://<org>.github.io/DAVE/;
• Local: python -m http.server in the repo root, then open http://localhost:8000/;
• The Hugging Face mirror also previews the sample audio per file.

A full training pair additionally carries s1.txt / s2.txt (Fun-ASR-Nano transcriptions used by the CER loss).

Training

bash run_train.sh — the final-stage configuration (native torch DDP, 7×A800, per-GPU batch 1); the curriculum stages are documented inside the script. The separation backbone, the PIT loss, and the WeSpeaker speaker extractor are bundled under src/models/, so no external separation repository is needed. External requirements: the synthesized corpus (src/synth/), the WeSpeaker cnceleb-resnet34-LM checkpoint (SPK_CKPT), and the Fun-ASR-Nano teacher (auto-downloaded, or set ASR_MODEL to a local directory). Data-list paths inside src/train.py are placeholders (/path/to/...); configure them for your environment. Weights resume from RESUME_CKPT; save interval via SAVE_EVERY.

Training Objective

A six-term loss, introduced as a curriculum (one term at a time, verified for no-regression before its weight is raised):

L = L_SI-SDR + λ_cer·L_CER + λ_spk·L_spk + λ_utmos·L_UTMOS + λ_stoi·L_STOI + λ_pesq·L_PESQ

Weights above are the final (rebalanced) configuration; earlier curriculum stages use smaller weights. Perceptual losses apply only to clean-reference samples.

Evaluation

reference/evaluation/ reproduces our seven-metric development-set measurement in two stages: bare_sep.py separates a fixed 200-scene sample with each checkpoint, and bare_judge.py scores the outputs with evaluation-aligned engines (SI-SDR / PESQ / STOI via torchmetrics; UTMOS / DNSMOS / ASR-CER / speaker cosine via the challenge Baseline's metric tracker). Both scripts depend on the challenge Baseline and the development data — see reference/README.md.

Data

The separation model is trained on synthetic two-speaker mixtures (each pair = five files: mix.wav, s1.wav, s2.wav, s1.txt, s2.txt, 16 kHz mono).

The training pool combines four subsets, three of which are reproducible with the scripts under src/synth/: RIR-simulated mixtures (synthesize.py), real-acoustic mixtures built from far-field MISP segments (synthesize_misp.py), and direct-sum mixtures with exact-zero residual (synthesize_pure.py). The fourth subset (a protocol anchor built from official challenge dev data) is intentionally not included; the pipeline runs without it. Transcripts for the CER loss are produced by transcribe.py (Fun-ASR-Nano) over the synthesized pair directories.

DAVE-Corpus (open subset) — 12,000 clean-provenance pairs — is released on Hugging Face (CC BY-SA 4.0): 8,000 RIR-simulated + 4,000 direct-sum pairs built exclusively from AliMeeting / AISHELL-4 speech and MUSAN noise with the pipeline in src/synth/.

The remaining subsets of the paper's training pool are not redistributable — they draw on MISP (non-commercial agreement) and official challenge dev data (challenge terms). For those, this repository releases the synthesis scripts, the per-pair manifests, and 20 example pairs (samples/, Apache-2.0 EchoSet subset); rebuild them by downloading the source corpora under their own licenses and running the synthesis. Full per-source license table and download guide: DATA_LICENSE.md.

Scope and Limitations

• Input format: 16 kHz mono, two-speaker mixtures (num_sources=2 is fixed); inference.py rejects other sample rates.
• Language: the released backbone is optimized for Chinese meeting speech (training corpora and the CER teacher are Chinese); other languages are untested.
• Separation only: inference.py outputs two anonymous streams — the s1/s2 order is arbitrary. Assigning streams to on-screen speakers requires face video and the attribution stage (reference/attribution/).
• Challenge-tuned: the enhancement chain and its thresholds were calibrated on the challenge evaluation; treat them as a reference design rather than defaults for other deployments.

Speaker-Attribution Models (external, cited)

Citation

@misc{zhou2026dave,
  title         = {{DAVE}: A Decoupled Audio-Visual Enhancement Framework for Real-World Speech Separation},
  author        = {Zhou, Wei and Ning, Wanyi and Guo, Yinshang and Fang, Qianxiao and Qian, Haitao and Li, Yingpeng},
  year          = {2026},
  eprint        = {2608.09288},
  archivePrefix = {arXiv},
  primaryClass  = {cs.SD}
}

Acknowledgements

Built on TIGER and the challenge baseline. Source-data attribution in DATA_LICENSE.md. Code and weights are Apache-2.0.

Contact

Questions and issues are welcome via GitHub Issues; for other inquiries contact [email protected].