WeSpeaker U³-xi

💡 WeSpeaker implementations for uncertainty-aware speaker recognition and verification, covering uncertainty estimation, speaker embedding learning, scoring, score normalization, and calibration.

Keywords: RecXi, NeurIPS 2023, speaker recognition, speaker verification, speaker embedding, WeSpeaker implementation, source code, ECAPA-TDNN, uncertainty estimation, uncertainty-aware speaker modeling.

Core idea · Getting started · U³-xi · Robust speaker modeling · Unified back-end · RecXi · Citation

📰 News

• [Sep 2026] 🎉 Our paper "$\mathcal{U}^3$-xi: Pushing the Boundaries of Speaker Recognition by Incorporating Uncertainty" has been accepted by IEEE Transactions on Audio, Speech, and Language Processing (TASLP)!

Core Idea

Editable draw.io source · Vector SVG

The goal is to estimate higher uncertainty when speech provides less reliable speaker evidence, such as under noise or reverberation. Frame-level precision estimates guide Bayesian pooling so reliable frames contribute more to the utterance representation. The model outputs both a speaker embedding and its associated uncertainty.

Conceptual illustration; the waveforms and distributions are schematic, not experimental measurements.

📚 Papers and Implementations

• $\mathcal{U}^3$-xi: Pushing the Boundaries of Speaker Recognition by Incorporating Uncertainty (accepted by IEEE TASLP) — official implementation of multi-view uncertainty estimation, uncertainty-aware training, and uncertainty-aware cosine scoring.
• Towards Robust Uncertainty-Aware Speaker Modeling — official implementation of inter- and intra-speaker-aware uncertainty modeling with AAM-Softmax, AM-Softmax, and SphereFace2. The UCDA method described in the paper is not included in this release.
• A Unified Uncertainty-Aware Back-End for Speaker Verification: Scoring, Normalization, and Calibration — official implementation of uncertainty-aware cosine scoring, UAS-Norm, and UQMF calibration.
• Disentangling Voice and Content with Self-Supervision for Speaker Recognition (RecXi, NeurIPS 2023) — an unofficial implementation and extension with parallel RecXi pooling, a multi-view Transformer precision estimator, and uncertainty-aware AAM-Softmax. See the RecXi implementation, training recipe, and results.

Getting Started

This project builds on WeSpeaker. Refer to the original WeSpeaker README for environment setup and the VoxCeleb recipe for data preparation. Use the configurations and scripts in this repository for the uncertainty-aware experiments described below.

• Main recipe: examples/voxceleb/v2/run.sh
• Model configurations: examples/voxceleb/v2/conf
• RecXi recipe: examples/voxceleb/v2/readme_recxi.md

Run recipe scripts from examples/voxceleb/v2 after configuring the dataset paths, GPU IDs, and experiment directory for your environment.

1. $\mathcal{U}^3$-xi: Pushing the Boundaries of Speaker Recognition by Incorporating Uncertainty

Junjie Li, Kong Aik Lee
The Hong Kong Polytechnic University

Email: junjie98.li@connect.polyu.hk

Key Highlights

U³-xi incorporates uncertainty into speaker embedding learning and verification. The implementation provides:

• Multi-view uncertainty estimation: a multi-view self-attention (MVA) module estimates the uncertainty of frame-level representations.
• Global uncertainty supervision: predicted uncertainty adjusts the softmax scale during training.
• Uncertainty-aware cosine scoring: embedding uncertainty is incorporated into verification scores.

Experiments

Training and evaluation recipes are available in examples/voxceleb/v2. The main implementation entry points are listed below.

• Configurations
  • ecapa_tdnn_u_cube.yaml
  • resnet_u_cube.yaml
  • redimnet_u_cube.yaml
• Models
  • ECAPA-TDNN
  • ResNet
  • ReDimNet
• Pooling layer: U_Cube_XI
  • Multi-view self-attention: MVA
• Projection layer: ArcMarginProduct_uncertainty
• Training executor: executor.py
• Training script: train.py. TorchScript compilation is disabled in this recipe because the MVA module is not compatible with it.

Results

The table reports equal error rate (EER, %) on the cleaned VoxCeleb1-O, VoxCeleb1-E, and VoxCeleb1-H trial lists. Lower values are better. LM denotes large-margin fine-tuning, AS-Norm denotes adaptive score normalization, and QMF denotes quality measure function calibration. ✓ and × indicate whether a component is enabled; rows with a blank model name use the model listed above; blank FLOPs values for named models were not reported.

Pretrained Models

These checkpoints were retrained, so their results may differ slightly from those reported in the paper.

• Voxceleb2_ECAPA-TDNN-512_u_cube
• Voxceleb2_ResNet34_u_cube
• Voxceleb2_ReDimNet-B2_u_cube
• Voxceleb2_Xi-ECAPA-TDNN-512

2. Towards Robust Uncertainty-Aware Speaker Modeling

This work addresses uncertainty estimation and calibration under domain shifts through two methods:

• Inter- and Intra-Speaker-Aware Uncertainty Softmax: incorporates inter-speaker separability and intra-speaker variability into uncertainty learning.
• Uncertainty-Calibrated Domain Adaptation (UCDA): addresses uncertainty miscalibration caused by domain mismatch.

Implementation scope: This release includes Inter- and Intra-Speaker-Aware Uncertainty Softmax only. UCDA is described in the paper but is not included in this repository.

Implementations

• UAAM-Softmax inter-intra: ArcMarginProduct_uncertainty_inter_intra
• USphereFace2 inter-intra: SphereFace2_uncertainty_Arcguide_inter_intra
• UAM-Softmax inter-intra: AddMarginProduct_uncertainty_inter_intra

Training note: train.py supports alpha scheduling for USphereFace2 inter-intra.

Results

The table reports EER (%) and minimum detection cost function (minDCF) values for in-domain evaluation on VoxCeleb1 and cross-domain evaluation on CNCeleb. Lower values are better. RI denotes relative improvement over the corresponding benchmark, and links in the loss column point to model checkpoints.

3. A Unified Uncertainty-Aware Back-End for Speaker Verification: Scoring, Normalization, and Calibration

This work carries embedding uncertainty through the speaker verification back-end, from similarity scoring to normalization and calibration. Each utterance is represented by a speaker embedding and its covariance estimate.

The unified pipeline includes three components:

• Uncertainty-aware cosine scoring: adjusts similarity scoring using embedding uncertainty.
• Uncertainty-aware AS-Norm (UAS-Norm): incorporates uncertainty into score normalization.
• Uncertainty-aware quality measure function calibration (UQMF): uses uncertainty-related features for score calibration.

Experiments

The VoxCeleb recipe includes uncertainty extraction, scoring, normalization, and calibration. The corresponding scripts are listed in pipeline order below.

• Uncertainty extraction

  • local/extract_vox_uncertainty.sh
  • tools/extract_embedding_uncertainty.sh
  • wespeaker/bin/extract_uncertainty.py

• Uncertainty-aware cosine scoring

  • local/score_uncertainty.sh
  • wespeaker/bin/score_uncertainty.py

• Uncertainty-aware AS-Norm (UAS-Norm)

  • local/score_norm_uncertainty.sh
  • tools/variance_mean.py
  • wespeaker/bin/score_norm_uncertainty.py

• Uncertainty-aware calibration (UQMF)

  • local/score_calibration_uncertainty.sh
  • wespeaker/bin/score_norm_forvox2.py
  • wespeaker/bin/score_calibration_uncertain.py

Results

The table reports EER (%) and minDCF on Vox1-O, Vox1-E, and Vox1-H. Lower values are better. Bold and underlined values denote the best and second-best results, respectively, within each architecture group. RI is the average relative improvement over the corresponding architecture-specific benchmark across the six VoxCeleb metrics. The suffixes -o and -u denote the original and uncertainty-aware variants, respectively; ①–③ identify the UAS-Norm components evaluated in the paper.

^† Results obtained directly from pretrained WeSpeaker models.

Citation

If you use this repository in your research, please cite the papers relevant to the methods you use. BibTeX entries are provided below; select an IEEE bibliography style in your manuscript if required.

• XI+: uncertainty supervision for robust speaker embeddings.
• U³-xi: multi-view uncertainty estimation and uncertainty-aware speaker modeling.
• Towards Robust Uncertainty-Aware Speaker Modeling: inter- and intra-speaker-aware uncertainty learning.
• A Unified Uncertainty-Aware Back-End: uncertainty-aware scoring, normalization, and calibration.

@inproceedings{li2026xiplus,
  author    = {Junjie Li and Kong Aik Lee and Duc-Tuan Truong and Tianchi Liu and Man-Wai Mak},
  title     = {{XI+}: Uncertainty Supervision for Robust Speaker Embedding},
  booktitle = {2026 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP)},
  year      = {2026},
  pages     = {18847--18851},
  doi       = {10.1109/ICASSP55912.2026.11463369}
}

@article{li2026u3xi,
  author  = {Junjie Li and Kong Aik Lee},
  title   = {{U3}-xi: Pushing the Boundaries of Speaker Recognition by Incorporating Uncertainty},
  journal = {arXiv preprint arXiv:2601.15719},
  year    = {2026},
  doi     = {10.48550/arXiv.2601.15719},
  url     = {https://arxiv.org/abs/2601.15719}
}

@article{li2026robust,
  author  = {Junjie Li and Yang Xiao and Kong Aik Lee},
  title   = {Towards Robust Uncertainty-Aware Speaker Modeling},
  journal = {arXiv preprint arXiv:2607.04937},
  year    = {2026},
  doi     = {10.48550/arXiv.2607.04937},
  url     = {https://arxiv.org/abs/2607.04937}
}

@article{li2026unified,
  author  = {Junjie Li and Kong Aik Lee},
  title   = {A Unified Uncertainty-Aware Back-End for Speaker Verification: Scoring, Normalization, and Calibration},
  journal = {arXiv preprint arXiv:2609.01221},
  year    = {2026},
  doi     = {10.48550/arXiv.2609.01221},
  url     = {https://arxiv.org/abs/2609.01221}
}

If you use the RecXi implementation, please also cite the original RecXi paper:

@inproceedings{liu2023disentangling,
  author    = {Tianchi Liu and Kong Aik Lee and Qiongqiong Wang and Haizhou Li},
  title     = {Disentangling Voice and Content with Self-Supervision for Speaker Recognition},
  booktitle = {Advances in Neural Information Processing Systems},
  volume    = {36},
  year      = {2023},
  url       = {https://proceedings.neurips.cc/paper_files/paper/2023/hash/9d276b0a087efdd2404f3295b26c24c1-Abstract-Conference.html}
}

Please also acknowledge the underlying toolkit using the WeSpeaker citations.