Audio Liveness Detection using Speaker Disentanglement

Major Technical Project (MTP) | IIT Mandi | 2024–2025
Shiven Patel | Guide: Dr. Padmanabhan

Overview

This project investigates audio liveness detection (anti-spoofing) on the ASVspoof 2019 and ASVspoof 2021 Physical Access (PA) benchmarks.

MTP 1 established strong baselines using large pre-trained audio models. MTP 2 explored speaker disentanglement and introduces a novel Direction Alignment technique for cross-domain generalisation.

Key Results

Novel Contribution: Direction Alignment

See direction_alignment/README.md for a full description.

In short: during early training on ASVspoof 2019, the bonafide–spoof direction vector in DaSHeng's embedding space becomes highly aligned (cosine similarity → 1) with the same direction computed on ASVspoof 2021.

Repository Structure

audio-liveness-detection/
├── baselines/                        Official ASVspoof 2021 baselines
│   └── 2021-main/                    SSL_Anti-spoofing reference codebase
│
├── W2V2 Fusion codebase/             ← Wav2Vec2 Layer Fusion experiments (MTP 2)
│   └── SSL_Anti-spoofing-main/       Wav2Vec2 + AASIST layer fusion implementation
│       ├── model.py                  Model architecture (Wav2Vec2 + AASIST backbone)
│       ├── main_SSL_LA.py            Training script — Logical Access track
│       ├── main_SSL_DF.py            Training script — Deepfake track
│       ├── Simplified_CM_solution.py Simplified countermeasure baseline
│       ├── data_utils_SSL.py         Dataset utilities for SSL models
│       ├── eval_metric_LA.py         EER/t-DCF evaluation — LA track
│       └── RawBoost.py               RawBoost data augmentation
│
├── models/                           Deep learning model definitions
│   ├── dasheng_classifier.py         DaSHeng + MLP head
│   ├── dasheng_ecapa_classifier.py   DaSHeng + ECAPA-TDNN fusion model
│   ├── ecapa_tdnn.py                 ECAPA-TDNN speaker encoder wrapper
│   └── AutoVC spk + class.py        AutoVC D_VECTOR encoder + classifier
│
├── training/                         Training scripts
│   ├── train_classifier_dasheng.py   DaSHeng-only classifier (control; 4.38% EER)
│   ├── train_classifier_dasheng_ecapa.py  DaSHeng + ECAPA-TDNN fusion (4.52% EER)
│   ├── train_dasheng.py              DaSHeng with direction alignment monitoring
│   ├── train_dasheng_ecapa.py        DaSHeng + ECAPA-TDNN (earlier version)
│   ├── train_msclap.py               MS-CLAP fine-tuning (5.3% EER)
│   ├── train_wav2vec2.py             Wav2Vec2 fine-tuning
│   ├── Train AutoVC spk.py           AutoVC speaker encoder training only
│   ├── train ecapa+classifer.py      Frozen ECAPA-TDNN + classifier head
│   └── Dasheng Augment.py            DaSHeng with aggressive data augmentation (25.8% EER on 2021)
│
├── evaluation/                       Evaluation and inference scripts
│   ├── calc_eer.py                   Compute EER from score files
│   ├── infer_dasheng.py              DaSHeng inference
│   ├── infer_msclap.py               MS-CLAP inference (2019)
│   ├── infer_msclap_2021.py          MS-CLAP inference (2021)
│   └── infer_wav2vec2.py             Wav2Vec2 inference
│
├── direction_alignment/              ← Novel contribution
│   ├── README.md                     Detailed description of the technique
│   ├── train_direction_alignment.py  Training with direction alignment monitoring
│   ├── test_direction_alignment.py   Cross-domain inference via KNN embedding shift
│   └── test_wav2vec2_direction.py    Direction alignment for Wav2Vec2
│
├── visualization/                    Visualisation and analysis scripts
│   ├── tsne_alignment_plot.py        t-SNE of aligned embeddings
│   ├── tsne_dasheng_speaker_enc.py   t-SNE of DaSHeng + speaker encoder embeddings
│   └── tsne_dataset_comparison.py   t-SNE comparison of 2019 vs 2021 domains
│
├── utils/                            Utility modules
│   ├── dataset.py                    ASVspoof dataset base class
│   └── data_utils.py                 Data loading utilities
│
├── data/                             Dataset directory (create and populate)
│   ├── ASVspoof2019_PA_train/
│   ├── ASVspoof2019_PA_dev/
│   ├── ASVspoof2019_PA_eval/
│   └── ASVspoof2021_PA_eval/
│
├── checkpoints/                      Trained model checkpoints (generated at runtime)
├── cache/                            Cached audio features (generated at runtime)
├── README.md
└── requirements.txt

Datasets

Download the required datasets from the official sources:

ASVspoof 2021 Physical Access (PA) Evaluation SubSet for small scale testing, download entire dataset seperately for full testing:

cd data
wget https://zenodo.org/records/4834716/files/ASVspoof2021_PA_eval_part00.tar.gz?download=1
wget https://www.asvspoof.org/asvspoof2021/PA-keys-full.tar.gz
wget https://datashare.ed.ac.uk/bitstream/handle/10283/3336/PA.zip?sequence=4&isAllowed=y

ASVspoof 2019 Physical Access (PA):

• ASVspoof 2019 — ~210k utterances, 107 speakers (PA track)

ASVspoof 2021:

• ASVspoof 2021 — ~990k utterances, 48 speakers

Use the official train split for training, dev for validation, eval for testing. Raw audio files are not stored in this repository. Extract all datasets into the data/ directory.

Requirements

pip install -r requirements.txt

Primary dependencies: PyTorch, torchaudio, Hugging Face Transformers, librosa, soundfile, pyroomacoustics.

Usage

Note on paths: The training scripts (train_classifier_*.py, Train AutoVC spk + class.py, train ecapa+classifer.py) currently use hardcoded absolute Windows paths at the top of each file. Edit the PROTOCOL_TRAIN, FLAC_TRAIN, PROTOCOL_DEV, FLAC_DEV, and (where applicable) local_sb_model_dir / DVECTOR_CKPT_PATH variables before running.

Train DaSHeng-only classifier (control, 4.38% EER on 2019):

python training/train_classifier_dasheng.py
# Edit PROTOCOL_TRAIN, FLAC_TRAIN, PROTOCOL_DEV, FLAC_DEV at top of file first

Train DaSHeng + ECAPA-TDNN fusion (4.52% EER on 2019):

python training/train_classifier_dasheng_ecapa.py
# Also requires local SpeechBrain ECAPA-TDNN weights — set local_sb_model_dir

Train AutoVC speaker encoder + classifier (18.4% EER on 2019):

python "training/Train AutoVC spk + class.py"
# Requires AutoVC D_VECTOR checkpoint (3000000-BL.ckpt) and model_bl.py
# Set DVECTOR_CKPT_PATH and AUTOVC_DIR at top of file

Train ECAPA-TDNN-only classifier:

python "training/train ecapa+classifer.py"
# Set local_sb_model_dir, PROTOCOL_2019_TRAIN, FLAC_2019_TRAIN at top of file

Train MS-CLAP:

python training/train_msclap.py

Train DaSHeng with direction alignment monitoring:

python training/train_dasheng.py \
    --protocol_2019 /path/to/ASVspoof2019_PA_cm_protocols/train.trn.txt \
    --flac_2019     /path/to/ASVspoof2019_PA_train/flac

Train Wav2Vec2 + AASIST layer fusion (layers 4, 7, 12):

cd "W2V2 Fusion codebase/SSL_Anti-spoofing-main"
python main_SSL_LA.py --track LA   # Logical Access track
# See W2V2 Fusion codebase/SSL_Anti-spoofing-main/README.md for full arguments

Evaluate on ASVspoof 2021 using direction alignment:

python direction_alignment/test_direction_alignment.py \
    --checkpoint    /path/to/checkpoint.pt \
    --protocol_2021 /path/to/ASVspoof2021_PA_eval.txt \
    --flac_2021     /path/to/ASVspoof2021_PA_eval/flac \
    --protocol_2019 /path/to/ASVspoof2019_PA_eval.txt \
    --flac_2019     /path/to/ASVspoof2019_PA_eval/flac

Calculate EER from a score file:

python evaluation/calc_eer.py --scores scores.txt

References

1. Kinnunen et al., "ASVspoof 2019," Computer Speech & Language, 2020. arXiv
2. Yi et al., "ASVspoof 2021," arXiv:2210.02437, 2022. arXiv
3. Qian et al., "AutoVC," ICML 2019.
4. Yu et al., "MS-CLAP," Microsoft Research, 2022.
5. Baevski et al., "wav2vec 2.0," NeurIPS 2020.
6. Jung et al., "AASIST," ICASSP 2022. arXiv
7. Liu et al., "Dasheng: Scaling up Audio Representations," 2024.
8. Desplanques