📢：Good news! 8000 hours of multi-label Wu dialect data are also available at ⭐WenetSpeech-Wu⭐.

📢：Good news! 10,000 hours of multi-label Chuan-Yu speech data are also available at ⭐WenetSpeech-Chuan⭐.

WenetSpeech-Yue: A Large-scale Cantonese Speech Corpus with Multi-dimensional Annotation

Longhao Li¹*, Zhao Guo¹*, Hongjie Chen², Yuhang Dai¹, Ziyu Zhang¹, Hongfei Xue¹, Tianlun Zuo¹, Chengyou Wang¹, Shuiyuan Wang¹, Xin Xu³, Hui Bu³, Jie Li², Jian Kang², Binbin Zhang⁴, Ruibin Yuan⁵, Ziya Zhou⁵, Wei Xue⁵, Lei Xie¹

¹ Audio, Speech and Language Processing Group (ASLP@NPU), Northwestern Polytechnical University
² Institute of Artificial Intelligence (TeleAI), China Telecom
³ Beijing AISHELL Technology Co., Ltd.
⁴ WeNet Open Source Community
⁵ Hong Kong University of Science and Technology

📑 Paper    |    🐙 GitHub    |    🤗 HuggingFace
🖥️ HuggingFace Space    |    🎤 Demo Page    |    💬 Contact Us

This is the official repository 👑 for the WenetSpeech-Yue dataset and the source code for WenetSpeech-Pipe speech data preprocessing pipeline.

📢 News and Updates

• 2026/01/07: 🛠️ We fixed several issues in the metadata. Please use the latest metadata for training.
• 2025/11/15: 🚀 We released Llasa-1B-Yue-Updated！ You can download the model weights from WSYue-TTS
• 2025/09/08: 🎉 The WenetSpeechYue dataset, featuring over 21,800 hours of Cantonese speech, is now available!

Download

• The WenetSpeech-Yue dataset is available at WenetSpeech-Yue.
• The WSYue-eval benchmark is available at WSYue-ASR-eval for ASR and WSYue-TTS-eval for TTS.
• The ASR models are available at WSYue-ASR.
• The TTS models are available at WSYue-TTS.

Dataset

WenetSpeech-Yue Overview

• Contains 21,800 hours of large-scale Cantonese speech corpus with rich annotations, the largest open-source resource for Cantonese speech research.
• Stores metadata in a single JSON file, including audio path, duration, text confidence, speaker identity, SNR, DNSMOS, age, gender, and character-level timestamps. Additional metadata tags may be added in the future.
• Covers ten domains: Storytelling, Entertainment, Drama, Culture, Vlog, Commentary, Education, Podcast, News, and Others.

Benchmark

To address the unique linguistic characteristics of Cantonese, we propose WSYue-eval, a comprehensive benchmark encompassing both Automatic Speech Recognition (ASR) and Text-to-Speech (TTS) tasks.

ASR Benchmark

We introduce WSYue-ASR-eval, a test set developed for Automatic Speech Recognition (ASR) as a key task in speech understanding. It features multi-round manual annotations including text transcripts, emotion, age, and gender labels. The set is divided into Short and Long subsets by audio duration to enable comprehensive evaluation across speech lengths. WSYue-ASR-eval also covers diverse real-world Cantonese scenarios, including code-switching and multi-domain conditions.

TTS Benchmark

We introduce WSYue-TTS-eval, a zero-shot Cantonese TTS benchmark with two subsets:

• Base: Contains 1,000 samples from Common Voice for evaluating real-world performance.
• Coverage: Combines manually curated and LLM-generated texts spanning multiple domains (e.g., daily life, news, entertainment, poetry) and incorporates diverse linguistic phenomena including polyphonic characters, tone sandhi, code-switching, proper nouns, and numerals.

ASR Leaderboard

ASR Inference

U2pp_Conformer_Yue

dir=u2pp_conformer_yue
decode_checkpoint=$dir/u2pp_conformer_yue.pt
test_set=path/to/test_set
test_result_dir=path/to/test_result_dir

python wenet/bin/recognize.py \
  --gpu 0 \
  --modes attention_rescoring \
  --config $dir/train.yaml \
  --test_data $test_set/data.list \
  --checkpoint $decode_checkpoint \
  --beam_size 10 \
  --batch_size 32 \
  --ctc_weight 0.5 \
  --result_dir $test_result_dir \
  --decoding_chunk_size -1

Whisper_Medium_Yue

dir=whisper_medium_yue
decode_checkpoint=$dir/whisper_medium_yue.pt
test_set=path/to/test_set
test_result_dir=path/to/test_result_dir

python wenet/bin/recognize.py \
  --gpu 0 \
  --modes attention \
  --config $dir/train.yaml \
  --test_data $test_set/data.list \
  --checkpoint $decode_checkpoint \
  --beam_size 10 \
  --batch_size 32 \
  --blank_penalty 0.0 \
  --ctc_weight 0.0 \
  --reverse_weight 0.0 \
  --result_dir $test_result_dir \
  --decoding_chunk_size -1

SenseVoice_Small_Yue

from funasr import AutoModel

model_dir = "sensevoice_small_yue"

model = AutoModel(
        model=model_path,
        device="cuda:0",
    )
res = model.generate(
    wav_path,
    cache={},
    language="yue",
    use_itn=True,
    batch_size=64,
)

TTS Inference

Install

Clone and install

• Clone the repo

git clone https://github.com/ASLP-lab/WenetSpeech-Yue.git
cd CosyVoice2-Yue

• Create Conda env:

conda create -n cosyvoice python=3.10
conda activate cosyvoice
# pynini is required by WeTextProcessing, use conda to install it as it can be executed on all platform.
conda install -y -c conda-forge pynini==2.1.5
pip install -r requirements.txt -i https://mirrors.aliyun.com/pypi/simple/ --trusted-host=mirrors.aliyun.com

Model download

from huggingface_hub import snapshot_download
snapshot_download('ASLP-lab/WSYue-TTS', local_dir='pretrained_models')

Usage

import sys
sys.path.append('third_party/Matcha-TTS')
from cosyvoice.cli.cosyvoice import CosyVoice, CosyVoice2
from cosyvoice.utils.file_utils import load_wav
import torchaudio
import opencc

# s2t
converter = opencc.OpenCC('s2t.json')

cosyvoice_base = CosyVoice2(
    'ASLP-lab/Cosyvoice2-Yue',
    load_jit=False, load_trt=False, load_vllm=False, fp16=False
)

cosyvoice_zjg = CosyVoice2(
    'ASLP-lab/Cosyvoice2-Yue-ZoengJyutGaai',
    load_jit=False, load_trt=False, load_vllm=False, fp16=False
)

prompt_speech_16k = load_wav('asset/sg_017_090.wav', 16000)

text = '收到朋友从远方寄嚟嘅生日礼物，嗰份意外嘅惊喜同埋深深嘅祝福令我心入面充满咗甜蜜嘅快乐，笑容好似花咁绽放。'
text = converter.convert(text)

for i, j in enumerate(cosyvoice_base.inference_instruct2(text, '用粤语说这句话', prompt_speech_16k, stream=False)):
    torchaudio.save('base_{}.wav'.format(i), j['tts_speech'], cosyvoice.sample_rate)

for i, j in enumerate(cosyvoice_zjg.inference_instruct2(text, '用粤语说这句话', prompt_speech_16k, stream=False)):
    torchaudio.save('zjg_{}.wav'.format(i), j['tts_speech'], cosyvoice.sample_rate)

WenetSpeech-Pipe

WenetSpeech-Pipe Overview:

Audio Collection

WenetSpeech-Pipe collects large-scale, in-the-wild speech recordings across diverse domains such as storytelling, drama, commentary, vlogs, food, entertainment, news, and education. These long recordings are segmented into short clips with VAD, yielding utterance-level data for transcription and quality evaluation.

Speaker Attribute Annotation

To enrich the dataset with speaker-level metadata for multi-speaker modeling and style-aware synthesis, WenetSpeech-Pipe includes a Speaker Attributes Annotation stage. Using pyannote toolkit for speaker diarization and Vox-Profile for age and gender estimation, each utterance-level segment is annotated with speaker identity, age, and gender, enabling supervised and style-controllable speech modeling.

Speech Quality Annotation

To support high-fidelity tasks such as TTS and voice conversion, WenetSpeech-Pipe integrates a comprehensive quality assessment stage. Each segment is evaluated by (i) Brouhaha for signal-to-noise ratio (SNR), (ii) DNSMOS for perceptual quality (MOS), and (iii) bandwidth detection for spectral coverage. These complementary measures yield structured annotations with quantitative scores and spectral references.

Automatic Speech Recognition

We select three models with the best performance on Cantonese to perform multi-system labeling: SenseVoice, TeleASR, and Whisper. For each audio file, we obtain the corresponding multi-system transcriptions.

Text Postprocessing

Each ASR transcription system produces outputs in different formats. To standardize these formats, we introduce a text post-processing module, which includes punctuation removal, traditional-to-simplified Chinese conversion, and text normalization. The detailed code can be found in text_postprocessing.py.

Recognizer Output Voting

Despite text postprocessing, ASR outputs still vary in lexical choice, segmentation, and phonetic representation. To obtain unified and accurate reference transcriptions, we adopt and extend the ROVER framework for Cantonese. Normalized transcriptions are first aligned using dynamic programming, after which a filtering module removes outlier outputs based on edit distance. Voting then selects the most frequent word at each aligned position, and the average voting frequency is recorded as an utterance-level confidence score. In parallel, we extend the voting mechanism to Cantonese pinyin by introducing a pronunciation-level confidence measure, further reinforcing phoneme consistency.

To further enhance transcription accuracy, we leverage Qwen3-4B for minimal, context-aware refinements of the consensus output. The LLM references all original ASR hypotheses and applies only essential corrections—such as grammar, lexical choice, or named entities—while preserving the integrity of the spoken content.

Contributors

Citation

Please cite our paper if you find this work useful:

@misc{li2025wenetspeechyuelargescalecantonesespeech,
      title={WenetSpeech-Yue: A Large-scale Cantonese Speech Corpus with Multi-dimensional Annotation}, 
      author={Longhao Li and Zhao Guo and Hongjie Chen and Yuhang Dai and Ziyu Zhang and Hongfei Xue and Tianlun Zuo and Chengyou Wang and Shuiyuan Wang and Jie Li and Xin Xu and Hui Bu and Binbin Zhang and Ruibin Yuan and Ziya Zhou and Wei Xue and Lei Xie},
      year={2025},
      eprint={2509.03959},
      archivePrefix={arXiv},
      primaryClass={cs.SD},
      url={https://arxiv.org/abs/2509.03959}, 
}

Contact

If you are interested in leaving a message to our research team, feel free to email lhli@mail.nwpu.edu.cn or gzhao@mail.nwpu.edu.cn.

Scan to join our WeChat discussion group