SpeechEditBench

Paper | Dataset | Evaluation Guide

SpeechEditBench is a bilingual multi-attribute benchmark for instruction-guided speech editing. Each example pairs source speech with a natural-language edit instruction, and the evaluator checks whether a system achieves the requested edit while preserving the expected lexical content.

The benchmark covers seven atomic editing tasks and compositional editing:

• content_editing
• speaker_editing
• emotion_editing
• style_editing
• prosody_editing
• paralinguistic_editing
• acoustic_editing
• compositional_editing

This repository contains the benchmark metadata, evaluation code, release manifest, and user-facing documentation. The full audio assets are distributed through Hugging Face.

Data

The v1.1 release contains 4,700 samples across 8 tasks and 5,400 audio files in the full data release.

The GitHub repository includes data/*/samples.jsonl for inspecting task schemas. Audio files are not stored in git. Download the released audio assets from Hugging Face:

git clone https://github.com/daxintan-cuhk/SpeechEditBench.git
cd SpeechEditBench

python scripts/download_hf_dataset.py \
  --repo-id DiscreteSpeech/SpeechEditBench \
  --revision v1.1

The download helper uses the local samples.jsonl files to fetch the benchmark metadata and referenced audio assets only, so it will not overwrite this repository's README or documentation and will not download extra unused audio files from the Hugging Face repository. Use --all-data to mirror every file under data/**, or --all-files to download the complete Hugging Face snapshot.

After download, the repository should contain:

data/<task_id>/samples.jsonl
data/<task_id>/audio/**

See docs/data.md for schema notes, the task input protocol, and release manifest details.

Installation

Create an environment with Python 3.11, which is the tested baseline for the v1.1 release, then install the Python dependencies. requirements.txt installs both the lightweight dataset helpers and the full evaluator stack:

conda create -n speecheditbench python=3.11
conda activate speecheditbench
pip install -r requirements.txt

For data download or metadata inspection only, use:

pip install -r requirements-core.txt

Several metrics require external evaluator models such as Whisper, Paraformer, WavLM, UTMOS, DNSMOS, and PANNs. These large files are not included in this repository. See docs/eval_models.md for the expected paths and upstream sources.

To check local evaluator readiness:

python scripts/check_eval_setup.py

The evaluator model download commands are listed in docs/eval_models.md.

Model Outputs

For each sample, save the edited audio as:

<output_dir>/<sample_id>.wav

The evaluator also accepts .flac and .mp3, and it can read outputs placed in an audio/ subdirectory:

<output_dir>/audio/<sample_id>.wav

For all-task evaluation, use:

<output_root>/<task_id>/<sample_id>.wav

or:

<output_root>/<task_id>/audio/<sample_id>.wav

In the standard benchmark setting, the model-visible inputs are the source audio_path, the natural-language instruction, and speaker reference audio only for speaker-editing tasks/components. Ground-truth anchor fields and acoustic target_reference_path entries are evaluator labels, not hidden model inputs.

Evaluation

Run one task:

python scripts/run_eval.py \
  --task content_editing \
  --output-dir outputs/my_model/content_editing \
  --model-name my_model

Run all tasks:

python scripts/run_eval.py \
  --task all \
  --output-root outputs/my_model \
  --model-name my_model

Runner messages default to Chinese. Use English runner messages with:

python scripts/run_eval.py \
  --task content_editing \
  --output-dir outputs/my_model/content_editing \
  --model-name my_model \
  --cli-lang en

Results are written under:

eval_results/<model_name>/<task_id>/<eval_set>/

See docs/evaluation.md for metric definitions, output layout, and task-specific notes.

Use --strict when you want CI-style failure semantics: missing outputs and per-sample evaluation errors will make the runner exit non-zero. Task-level preflight errors, task exceptions, and zero evaluated outputs always exit non-zero.

Version

The current release manifest is:

release_manifests/v1.1/dataset_manifest.json

License

The SpeechEditBench code, documentation, metadata, and benchmark assets authored by the SpeechEditBench contributors are released under the Apache License 2.0. See LICENSE.

The benchmark is derived from multiple upstream speech corpora. Users are responsible for complying with the applicable upstream dataset licenses and terms when using the released audio assets.

Citation

If you use SpeechEditBench, please cite:

@article{zhang2026speecheditbench,
  title={SpeechEditBench: A Bilingual Multi-Attribute Benchmark for Instruction-Guided Speech Editing},
  author={Zhang, Hanlin and Tan, Daxin and Tao, Dehua and Chen, Xiao and Tan, Haochen and Song, Linqi},
  journal={arXiv preprint arXiv:2606.01804},
  year={2026}
}