RVCBench — Voice Cloning Benchmark

RVCBench is the first large-scale benchmark for voice cloning robustness, voice clone evaluation, speaker privacy, and audio deepfake protection — covering 27 TTS/VC models, 10 datasets, and 5 audio protection methods.

RVCBench provides a unified, reproducible pipeline covering the full attack-defense cycle: source-audio protection, zero-shot or fine-tuning voice cloning, optional denoising, and evaluation of speaker similarity, intelligibility, perceptual quality, and runtime.

At a glance, this release supports 27 VC/TTS adversary models, 5 audio protection methods, 10 public benchmark dataset configurations, and both fidelity and generation-quality metrics.

[!NOTE] Paper vs. codebase. The arXiv v2 paper reports results for 18 models across 18 robustness evaluations, 225 speakers, and 14,370 utterances. This repository is under active development and currently supports 27 models; treat any count beyond the paper's 18 as codebase-only until covered by a future paper revision.

Canonical resources: website · paper · Hugging Face dataset · interactive demo · quickstart notebooks · model environments · citation

Contents: Results · Models · Getting Started · Quickstart · Full Pipeline · Data & Checkpoints · Citation

---

Overview

Voice cloning technology poses a growing threat to speaker privacy. Audio protection methods—such as adversarial perturbations—aim to make cloned speech recognizably degraded, but their effectiveness varies widely across different VC models and datasets. RVCBench closes this gap by offering a single framework that:

• applies protection algorithms to source audio (SafeSpeech, Enkidu, EM, Gaussian noise, spectral perturbations),
• runs a broad suite of zero-shot and fine-tuning VC adversaries on clean or protected inputs,
• optionally denoises protected audio and re-evaluates,
• computes standardised fidelity and generation-quality metrics with bootstrap confidence intervals.

RVCBench is intended for researchers and engineers working on voice cloning benchmarks, audio deepfake robustness, speaker verification resilience, anti-spoofing, synthetic speech detection, TTS safety, and privacy-preserving speech generation.

Why RVCBench

Most protection papers evaluate against one or two surrogate VC models on one dataset — a defense that looks strong there can fail completely against a model or language it was never tested on. RVCBench exists to close that gap:

If you're deciding whether a protection method actually generalises — or whether a new TTS model breaks existing defenses — this is the benchmark to run it against.

---

Benchmark Results

[!NOTE] Metric guide — SIM: speaker cosine similarity ↑ · WER: word error rate ↓ · MOS: SpeechMOS perceptual score ↑ · MCD: mel cepstral distortion ↓ · RTF: real-time factor (< 1 = faster-than-real-time) ↓ · SVA: speaker verification accuracy ↑ · Emo: emotion match rate ↑

Bold marks the best value per column. All results on clean (unprotected) prompts, averaged over the full speaker set for each dataset.

Leaderboard — LibriTTS

Protection Robustness — SIM on LibriTTS

Speaker similarity under each audio protection method. Models sorted by clean SIM. A larger drop from Clean indicates more effective protection. Bold marks the lowest protected SIM per column (most effectively protected model per method).

Cross-Dataset Generalisation — SIM across all 10 datasets (click to expand)

Speaker similarity (SIM) on clean prompts across all benchmark datasets. — indicates the model was not evaluated on that dataset.

Model	LibriTTS	VCTK	Multi-spk	Long	AISHELL	French	Bilingual	BG-clean	BG-noise	Hallucin.
Qwen3-TTS	0.614	0.618	0.495	0.561	0.721	0.536	0.673	0.689	0.572	0.515
IndexTTS	0.606	0.567	0.473	0.775	0.721	0.397	0.673	0.589	0.528	0.529
CosyVoice 2	0.602	0.582	0.448	0.530	0.717	0.378	0.653	0.626	0.515	0.518
ZipVoice	0.579	0.554	0.531	0.729	0.712	0.363	0.322	0.625	0.462	0.509
MaskGCT	0.570	0.555	0.431	0.194	0.674	0.494	—	0.610	0.487	0.499
GLM-TTS	0.570	0.573	0.445	0.757	0.690	0.398	0.657	0.622	0.528	0.533
F5-TTS	0.559	0.537	0.507	0.607	0.696	0.304	0.653	0.582	0.414	0.455
Higgs Audio	0.559	0.516	0.418	0.520	0.581	0.349	0.543	0.592	0.421	0.425
MGM-Omni	0.539	0.447	0.370	0.442	0.713	0.227	0.630	0.523	0.332	0.396
PlayDiffusion	0.506	0.426	0.360	0.637	0.441	0.283	0.465	0.433	0.305	0.408
MOSS-TTSD	0.492	0.440	0.379	0.644	0.437	0.327	0.471	0.494	0.488	0.416
VibeVoice	0.480	0.436	0.348	0.625	0.564	0.343	0.531	0.513	0.364	0.408
FishSpeech	0.472	0.430	0.383	0.572	0.611	0.374	0.566	0.495	0.387	0.351
XTTS-v2	0.454	0.454	0.328	0.613	0.569	0.445	0.506	0.546	0.394	0.488
SparkTTS	0.408	0.532	0.228	0.345	0.569	0.164	0.480	0.588	0.332	0.336
OZSpeech	0.388	0.253	0.271	—	—	0.109	—	0.272	0.164	0.281
OpenVoice V2	0.244	0.392	0.192	0.278	0.431	0.271	0.298	0.484	0.358	0.365
StyleTTS 2	0.228	0.236	0.162	—	—	—	0.213	0.196	0.166	0.184

---

Supported Models

Voice Cloning Adversaries (Zero-Shot OTS)

RVCBench currently includes wrappers or configs for 32 VC/TTS adversary models:

Protection Methods

RVCBench currently supports 5 audio protection methods:

Supported Datasets

The public Hugging Face dataset release exposes 10 benchmark dataset configurations:

Evaluated Metrics

RVCBench reports fidelity metrics for protection/denoising runs and generation-quality metrics for VC/TTS runs.

---

Getting Started

Requirements

• Python 3.9+
• PyTorch ≥ 2.0 with CUDA
• hydra-core, omegaconf, pandas, pyarrow, soundfile, librosa
• Model-specific packages. Many supported VC/TTS models need mutually incompatible dependency stacks, so launch each model from its matching environment in envs/; see docs/model_environments.md.

Installation

git clone https://github.com/Nanboy-Ronan/RVCBench.git
cd RVCBench
pip install hydra-core omegaconf pandas pyarrow soundfile librosa \
            jiwer openai-whisper pymcd huggingface_hub

For model-specific runs, prefer the checked-in Conda environment files:

cd envs
conda env create -f qwen3-tts.yml
conda activate qwen3
cd ..

Model checkpoints and third-party inference code are not bundled. Download instructions are provided in the Data & Checkpoints section.

---

Quickstart Path

[!TIP] If you only want to evaluate a single model, skip directly to Running Specific VC Models — you do not need to download every checkpoint bundle.

Use this path when you want to run a small, end-to-end example with automatic data download. All outputs — data, generated audio, and metrics — are written inside the repository directory. The Qwen3-TTS examples assume the qwen3 environment is active and the Qwen checkpoint is available from Hugging Face or a local path.

Fastest quickstart

Simplest voice-cloning run:

conda activate qwen3
python scripts/run_qwen3tts_quickstart.py --max-samples 5

Protection plus voice-cloning run:

conda activate qwen3
python scripts/run_protect_qwen3tts_quickstart.py --max-samples 5

Both commands download the selected LibriTTS speaker data from Nanboy/RVCBench unless --no-hf-download is passed.

Quickstart examples

Three end-to-end examples are provided as both Jupyter notebooks and standalone Python scripts.

Model setup for the quickstarts, including gated downloads and exact commands: docs/quickstart_model_setup.md

Additional script commands

conda activate qwen3

# Zero-shot voice cloning only
python scripts/run_qwen3tts_quickstart.py                    # auto-downloads data from HF
python scripts/run_qwen3tts_quickstart.py \
    --no-hf-download --speaker-id 1089 --max-samples 10
python scripts/run_qwen3tts_quickstart.py \
    --qwen-checkpoint-path checkpoints/Qwen3-TTS-12Hz-1.7B-Base

# Protection + voice clone attack (two-step pipeline)
python scripts/run_protect_qwen3tts_quickstart.py            # Gaussian noise protection (no checkpoints needed)
python scripts/run_protect_qwen3tts_quickstart.py \
    --protect-config safespeech_on_libritts                    # SafeSpeech (requires surrogate-model checkpoints)

# FishSpeech zero-shot voice cloning
python scripts/run_fishspeech_quickstart.py

# Fish Audio S2 zero-shot voice cloning
python scripts/run_fishspeech_s2_quickstart.py

# Smoke-test the notebook companion scripts against local HF-formatted data only
python scripts/validate_quickstarts.py

Pass --help to either script for the full list of options.

---

Full Benchmark Path

Use this path when you want to run the configurable benchmark entry points directly. All full benchmark entry points use Hydra for configuration, and config values can be overridden on the command line.

Experiment flow

source audio → [protection] → [denoising] → voice cloning → evaluation

1. Apply protection and evaluate fidelity

python run_protect.py --config-name safespeech_on_libritts

2. Run zero-shot voice cloning on clean prompts

# Using a specific model and dataset
python run_vc.py --config-name ots_vc/clean/libritts/qwen3_tts_ots

# With command-line overrides
python run_vc.py --config-name ots_vc/clean/libritts/qwen3_tts_ots \
    adversary.max_samples=50 dataset.speaker_id=1089

3. Run voice cloning on protected prompts

python run_vc_protect.py --config-name ots_vc/protection/safespeech/ozspeech_ots \
    protected_audio_dir=results/safespeech_on_libritts/<timestamp>/protected_audio

4. Optionally denoise protected audio and re-evaluate

python run_denoiser.py --config-name denoise/denoiser_dns64_on_protected_libritts_spec

---

Running Specific VC Models

The zero-shot VC configs are under configs/ots_vc/clean/. All LibriTTS model integrations now use the canonical configs/ots_vc/clean/libritts/ directory.

Before launching a model, activate the matching environment from envs/. See docs/model_environments.md for the full map.

If You Only Want One Model

You do not need to download every checkpoint bundle. For a single model, the workflow is:

1. Pick the Hydra config for that model under configs/ots_vc/clean/....
2. Create and activate that model's environment from envs/.
3. Download or install only that model's runtime and checkpoints.
4. Point any local paths with Hydra overrides such as adversary.code_path=... or adversary.checkpoint_path=....
5. Run run_vc.py against the dataset/config you want.

Generic pattern:

python run_vc.py --config-name <model_config> \
  dataset.speaker_id=<speaker_id> \
  adversary.max_samples=<n>

If the model needs a local repo checkout or checkpoint directory:

python run_vc.py --config-name <model_config> \
  dataset.speaker_id=<speaker_id> \
  adversary.code_path=/path/to/model_repo \
  adversary.checkpoint_path=/path/to/checkpoint_or_hf_id

Concrete examples:

# Qwen3-TTS only
python -m pip install -U qwen-tts
huggingface-cli download Qwen/Qwen3-TTS-12Hz-1.7B-Base \
  --local-dir checkpoints/Qwen3-TTS-12Hz-1.7B-Base
python run_vc.py --config-name ots_vc/clean/libritts/qwen3_tts_ots \
  dataset.speaker_id=1089 \
  adversary.checkpoint_path=checkpoints/Qwen3-TTS-12Hz-1.7B-Base

# FishSpeech only
git clone https://github.com/fishaudio/fish-speech.git checkpoints/fish_speech
huggingface-cli download fishaudio/s1-mini \
  --local-dir checkpoints/fish_speech/openaudio-s1-mini
python run_vc.py --config-name ots_vc/clean/vctk/fishspeech_ots \
  dataset.speaker_id=p226 \
  adversary.code_path=checkpoints/fish_speech \
  adversary.llama_checkpoint_path=checkpoints/fish_speech/openaudio-s1-mini \
  adversary.decoder_checkpoint_path=checkpoints/fish_speech/openaudio-s1-mini/codec.pth

For the full command list for the quickstart models, see docs/quickstart_model_setup.md.

Additional Examples

# FireRedTTS-2
python run_vc.py --config-name ots_vc/clean/libritts/fireredtts2_ots

# VoxCPM
python run_vc.py --config-name ots_vc/clean/libritts/voxcpm_ots

# dots.tts (conda env: dots-tts)
python run_vc.py --config-name ots_vc/clean/libritts/dots_tts_ots device=cuda:<gpu>

# MOSS-TTS (conda env: moss-tts)
python run_vc.py --config-name ots_vc/clean/libritts/moss_tts_ots device=cuda:<gpu>

# ZONOS2 (uv-managed env; run with its own interpreter, GPU pinned via CUDA_VISIBLE_DEVICES)
CUDA_VISIBLE_DEVICES=<gpu> checkpoints/ZONOS2-repo/.venv/bin/python run_vc.py \
  --config-name ots_vc/clean/libritts/zonos2_ots

Model-specific setup notes

• Qwen3-TTS can use either the Hugging Face model ID Qwen/Qwen3-TTS-12Hz-1.7B-Base directly or a local directory passed via adversary.checkpoint_path=.... It also needs the qwen-tts Python package.
• FishSpeech needs both a local checkout of fishaudio/fish-speech and the fishaudio/s1-mini checkpoint directory. Pass them with adversary.code_path=..., adversary.llama_checkpoint_path=..., and adversary.decoder_checkpoint_path=....
• Fish Audio S2 (paper) reuses the same fishaudio/fish-speech checkout as FishSpeech, pointed at the fishaudio/s2-pro checkpoint instead of s1-mini. S2 uses a different dual-AR decoder architecture than S1; the wrapper defaults decoder_config_name to modded_dac_vq as a best-effort setting that has not been validated against a downloaded s2-pro checkpoint — see docs/quickstart_model_setup.md.
• FireRedTTS-2 expects a local upstream checkout at checkpoints/FireRedTTS2 and pretrained weights under checkpoints/FireRedTTS2/pretrained_models/FireRedTTS2 by default.
• VoxCPM defaults to the Hugging Face model ID openbmb/VoxCPM2. If you want to force offline/local loading, override adversary.local_files_only=true and optionally set adversary.cache_dir=/path/to/cache.
• dots.tts (rednote-hilab/dots.tts-soar, pip-installable) needs its own env (envs/dots-tts.yml, see the file for the exact install order — it requires torch>=2.8.0, newer than the repo-wide requirements.txt pin). device: cuda:N is honoured.
• MOSS-TTS (OpenMOSS-Team/MOSS-TTS-v1.5, plain transformers + trust_remote_code) also needs its own env (envs/moss-tts.yml; see the file for the sequential pip-install order — installing everything in one shot fails to resolve).
• ZONOS2 (Zyphra/ZONOS2) is not pip-installable; it's a uv-managed project (see envs/zonos2.yml for the clone + uv sync setup steps). Its TTSLLM scheduler ignores device: cuda:N — pin the GPU with CUDA_VISIBLE_DEVICES instead — and it pre-allocates a large KV cache (~55GB on an 80GB A100), so run it alone on its GPU.
• dots.tts, ZONOS2, and MOSS-TTS each route the sample's target_language into the model runtime, which matters for AISHELL, French, and mixed-direction cross-lingual configs.
• Fish Audio S2 (local API server) and Higgs TTS 3 talk to a locally running inference server instead of loading weights in-process — see Server-backed models below.
• All model-specific paths and generation knobs can be overridden at launch time with Hydra, for example: adversary.code_path=/path/to/model_repo or adversary.max_samples=20.

Server-backed models: Fish Audio S2 and Higgs TTS 3

Unlike the other adversaries, these two don't load weights in-process — they call a local HTTP server (adversary.endpoint_url in the config), so you start the server first and then point run_vc.py at it.

Fish Audio S2 (fishaudio/s2-pro, served via Fish Speech's /v1/tts API):

git clone https://github.com/fishaudio/fish-speech.git checkpoints/fish_speech
conda env create -f envs/fish-speech-s2.yml
conda activate fish-speech-s2
uv pip install 'numba==0.63.1' 'llvmlite==0.46.0'
cd checkpoints/fish_speech
uv pip install -e '.[cu126]'
uv pip install 'protobuf>=6.31.1,<7'
cd ../..
hf download fishaudio/s2-pro --local-dir checkpoints/s2-pro
CUDA_VISIBLE_DEVICES=<gpu> python checkpoints/fish_speech/tools/api_server.py \
  --llama-checkpoint-path checkpoints/s2-pro \
  --decoder-checkpoint-path checkpoints/s2-pro/codec.pth \
  --listen 0.0.0.0:8001 --half
# in another shell, from the benchmark (audiobench) env — needs the `ormsgpack` dep in requirements.txt:
python run_vc.py --config-name ots_vc/clean/libritts/fish_audio_s2_ots

Higgs TTS 3 (bosonai/higgs-tts-3-4b, served via vLLM-Omni's OpenAI-compatible /v1/audio/speech API):

conda env create -f envs/vllm-omni-cu129.yml
conda activate vllm-omni-cu129
uv pip install --torch-backend=cu129 --extra-index-url https://wheels.vllm.ai/0.24.0/cu129 vllm==0.24.0
uv pip install --torch-backend=cu129 vllm-omni==0.24.0
hf download bosonai/higgs-tts-3-4b
CUDA_VISIBLE_DEVICES=<gpu> vllm-omni serve bosonai/higgs-tts-3-4b \
  --host 0.0.0.0 --port 8000 --trust-remote-code --omni \
  --allowed-local-media-path "$(pwd)"
# in another shell, from the benchmark env:
python run_vc.py --config-name ots_vc/clean/libritts/higgs_tts3_ots

Both server configs default device: cpu at the top level (the client-side process does no GPU work) with evaluation.device: cuda:0 for the metrics pass; point the server itself at a GPU via CUDA_VISIBLE_DEVICES as shown above.

Known metric gaps in the dots-tts, moss-tts, and zonos2 envs (env version conflicts, not model properties — see the emotion_pairs / speechmos_pairs fields in each run's metrics.json):

• dots-tts, moss-tts: emotion_pairs: 0 — the bundled speechbrain emotion recognizer needs AutoModelWithLMHead, which is removed in the transformers version these models require.
• zonos2: speechmos_pairs: 0, emotion_pairs: 0 — torchcodec fails to load its native library against this env's torch/ffmpeg combination, so the torchaudio.load_with_torchcodec path used for those two metrics fails. WER, MCD, SIM, DNSMOS, and DNSMOS-C are unaffected.

Example overrides

python run_vc.py --config-name ots_vc/clean/libritts/fireredtts2_ots \
    adversary.max_samples=20 \
    dataset.speaker_id=1089

python run_vc.py --config-name ots_vc/clean/libritts/voxcpm_ots \
    adversary.local_files_only=true \
    adversary.cache_dir=/path/to/hf-cache

---

Data & Checkpoints

Benchmark dataset

The benchmark dataset is publicly available on Hugging Face. The data loader fetches it automatically when use_hf_dataset: true (the default in all dataset configs).

👉 Nanboy/RVCBench on Hugging Face

A static snapshot is also available for offline use:

👉 Download via Google Drive

Model checkpoints

Each VC adversary requires its own inference code and pretrained weights. Clone the relevant repository into checkpoints/ and verify the paths in the corresponding config under configs/.

A bundled archive with all supported model code and checkpoints is available here:

👉 Download pretrained checkpoints (~58 GB)

If you only need a single model, cloning just that repository is the fastest option.

---

Dataset Format

Each dataset follows a canonical layout:

data/<dataset>/
├── audios/
│   └── <speaker_id>/*.wav
├── filelists/          # legacy per-speaker JSON manifests (kept for compatibility)
└── metadata.parquet    # canonical manifest used by all loaders

metadata.parquet stores one row per benchmark pair with the following columns:

Training-oriented phoneme and alignment annotations (prompt_phonemes, prompt_tone, prompt_word2ph, and their target_* counterparts) are preserved when available. Dataset-specific fields (e.g., spam_type in robotcall) are carried as additional columns.

To rebuild canonical manifests from legacy per-speaker JSON files:

python src/datasets/build_canonical_manifests.py --force

Dataset selection and speaker_id filtering continue to work through configs/dataset/ as before.

---

Repository Structure

RVCBench/
├── run_vc.py                  # voice cloning on clean prompts
├── run_protect.py             # apply protection + fidelity evaluation
├── run_vc_protect.py          # voice cloning on protected prompts
├── run_denoiser.py            # denoise protected audio + re-evaluate
├── configs/
│   ├── dataset/               # dataset configs (root_path, sampling_rate, …)
│   ├── model/                 # surrogate model configs for protection
│   ├── ots_vc/                # zero-shot VC configs (clean / protected)
│   └── denoise/               # denoiser configs
├── src/
│   ├── adversary/             # VC adversary wrappers
│   ├── protection/            # protection algorithm implementations
│   ├── datasets/              # dataset loaders and manifest utilities
│   ├── evaluation/            # fidelity and generation-quality metrics
│   ├── models/                # model generator wrappers
│   └── workflows/             # end-to-end pipeline orchestration
├── notebooks/                 # quickstart notebooks and runnable example scripts
└── data/                      # local dataset folders (populated at runtime)

---

Outputs

Each run writes a timestamped directory under results/:

results/<run_name>/<timestamp>/
├── generated_audio/            # cloned audio files (VC runs)
├── protected_audio/            # perturbed audio files (protection runs)
├── perturbed_noise/            # saved perturbation tensors
├── generation_sample_metrics.csv
├── fidelity_sample_metrics.csv
└── metrics.json                # aggregated metrics with bootstrap CIs

metrics.json contains fidelity metrics (SNR, STOI, MCD, WER, MOS, speaker similarity) for protection and denoising runs, and generation-quality metrics (MCD, WER, speaker similarity, MOS, emotion match rate, RTF) for VC runs.

---

Configuration

All configs use Hydra. Any field can be overridden from the command line:

python run_vc.py --config-name ots_vc/clean/libritts/qwen3_tts_ots \
    device=cuda:1 \
    adversary.max_samples=100 \
    dataset.speaker_id=1089 \
    dataset.use_hf_dataset=false \
    dataset.root_path=/path/to/local/data

Key config locations:

---

Contributing

Contributions are welcome. Areas of particular interest include:

• New protection or defense methods
• New VC adversary wrappers
• Dataset adapters and additional evaluation metrics
• Reproducibility and documentation improvements

Please open an issue or pull request on GitHub. For questions, contact:

ruinanjin@alumni.ubc.ca

---

Citation

If you use RVCBench in your research, please cite:

@article{jin2026rvcbench,
  title   = {RVCBench: Benchmarking the Robustness of Voice Cloning Across Modern Audio Generation Models},
  author  = {Jin, Ruinan and Liao, Xinting and Yu, Hanlin and Pandya, Deval and Li, Xiaoxiao},
  journal = {arXiv preprint arXiv:2602.00443},
  year    = {2026}
}

License

See LICENSE.