FastMSS: Fast Multi-Speaker Simulation

FastMSS is a toolkit for generating realistic multi-speaker meeting scenarios using room impulse response (RIR) simulation and HMM-based turn-taking models. It can dump per-speaker anechoic and reverberant reference signals, making it suitable for training speech separation and enhancement models in addition to speaker-attributed ASR and speaker diarization systems.

Demo

A simulated noisy + reverberant meeting generated by FastMSS (click to play on GitHub):

https://github.com/user-attachments/files/26752329/2efd6b4a-8fda-463e-8829-8cf885960f3d.wav

Speed

Generation throughput (simulated hours per minute) as a function of the number of parallel workers on a DGX node. FastMSS scales near-linearly and is significantly faster than MMS-MSG and NeMo simulators.

Features

• HMM-based turn-taking — models 4 transition types (turn hold, turn switch, interruption, backchannel) with configurable probabilities; can fit transitions to real corpora (AMI, CallHome, NOTSOFAR-1) or use flat priors
• Overlap control — boost overlap factor to increase/decrease interruption and backchannel rates
• Room simulation — RIR generation via pyroomacoustics with configurable room geometry and RT60
• Per-speaker outputs — optionally save individual speaker streams and anechoic references for separation/enhancement training
• Noise augmentation — add real noise (e.g. WHAM, MUSAN) at configurable SNR ranges
• Lhotse integration — reads source speech from lhotse CutSets with word-level alignments; outputs lhotse-compatible manifests
• RTTM + NeMo manifests — optionally generate RTTM files and NeMo-style diarization manifests

Installation

git clone https://github.com/popcornell/FastMSS.git
cd FastMSS
pip install -e .

Project Structure

FastMSS/
├── fastmss/                     # Core simulation library
│   ├── simulator.py             # Meeting generation engine
│   ├── hmm_turn_taking.py       # HMM turn-taking model
│   ├── rirsimulator.py          # RIR generation via pyroomacoustics
│   └── utils.py                 # Audio splitting, crossfading utilities
├── recipes/
│   ├── sim.py                   # Unified simulation pipeline
│   ├── default.yaml             # Default config with sensible defaults
│   └── paper/                   # Configs to reproduce paper experiments
│       ├── ts_asr/              # TS-ASR experiment configs (Table 1 & 2)
│       └── diarization/         # Diarization experiment configs
├── preprocessing/
│   ├── download_wham.sh         # Download WHAM noise dataset
│   ├── download_otospeech.sh    # Download + prepare otoSpeech dataset
│   └── resample_folder.py       # General audio resampling utility
├── setup.py
└── requirements.txt

Quick Start

1. Prepare source data

FastMSS takes single-speaker audio in lhotse format as input. You can use any speech corpus — here are two common options:

# Option A: LibriSpeech (via lhotse)
lhotse download librispeech --full /path/to/librispeech
lhotse prepare librispeech /path/to/librispeech /path/to/manifests

# Option B: otoSpeech (from HuggingFace)
bash preprocessing/download_otospeech.sh /path/to/manifests /path/to/otospeech

# Optional: download WHAM noise for augmentation
bash preprocessing/download_wham.sh /path/to/wham

2. Generate meetings

All paths are set via Hydra CLI overrides — no config file editing needed:

# Basic: 1000 meetings, 2-4 speakers, 60s each, flat turn-taking priors
python recipes/sim.py \
    output_dir=/path/to/output \
    manifest_dir=/path/to/manifests \
    n_meetings=1000 \
    duration=60 \
    n_jobs=16

# With noise + reverberation
python recipes/sim.py \
    output_dir=/path/to/output \
    manifest_dir=/path/to/manifests \
    noise_folders=[/path/to/wham] \
    add_noise=true \
    reverberate=true \
    n_jobs=16

# Save per-speaker anechoic references (for separation/enhancement)
python recipes/sim.py \
    output_dir=/path/to/output \
    manifest_dir=/path/to/manifests \
    save_spk=true \
    save_anechoic=true \
    reverberate=true \
    n_jobs=16

# Generate RTTM + NeMo manifests (for diarization)
python recipes/sim.py \
    output_dir=/path/to/output \
    manifest_dir=/path/to/manifests \
    save_rttm=true \
    save_nemo_manifest=true \
    n_jobs=16

Key parameters

General:

Augmentation (all independently toggleable):

Output options:

Turn-taking:

Turn-Taking Model

FastMSS uses a 4-state HMM to model conversational dynamics. At each turn boundary, the model samples one of:

Transition probabilities can be set to uniform (flat), fitted to a target corpus via hmm_fit_transitions_to, or manually specified via hmm_params.p_ind. The boost_overlap_factor parameter scales the interruption and backchannel probabilities to increase or decrease overall overlap.

Reproducing Paper Experiments

The recipes/paper/ directory contains the exact configs used in the Interspeech 2026 paper. To run them, override the config name:

# TS-ASR Table 1: flat turn-taking priors
python recipes/sim.py \
    --config-name paper/ts_asr/table1/flat \
    output_dir=/path/to/output \
    manifest_dir=/path/to/manifests

# Diarization: 2-speaker clean with overlap boosting
python recipes/sim.py \
    --config-name paper/diarization/clean/librispeech_2spk \
    output_dir=/path/to/output \
    manifest_dir=/path/to/manifests

Citation

If you use FastMSS in your research, please cite:

@unpublished{polok2026mindthegap,
    author = {Polok, Alexander and Medennikov, Ivan and Watanabe, Shinji and {\v{C}}ernock{\'y}, Jan and Burget, Luk{\'a}{\v{s}} and Cornell, Samuele},
    title = {Mind the Gap: Impact of Synthetic Conversational Data on Multi-Talker {ASR} and Speaker Diarization},
    note = {Submitted to INTERSPEECH},
    year = {2026}
}