README
StyleStream
A complete PyTorch reimplementation of StyleStream: Real-Time Zero-Shot Voice Style Conversion (arXiv:2602.20113).
Provenance
This is an independent, from-scratch reimplementation written from the published paper alone. No code from the original authors was used, viewed, or adapted while building it. Every module here was derived by reading the paper and reconstructing the described architecture in PyTorch.
The timeline is verifiable in this repository's git history: the style encoder landed 2026-03-16, roughly three months before the original authors published their model code (2026-06-11). Where this implementation differs from theirs, the difference is an interpretation gap, not a deviation from a reference that was available at the time.
The history explains itself if you read it. The first few commits are authored by the original authors, because this repository began in February 2026 as a fork of theirs — back when that repository held nothing but a project page: a
LICENSE, aREADME.md, anddocs/index.html. Not one.pyfile. There was no model code to copy, because none had been published. Everything understylestream/was written afterwards, from the paper. The fork link has since been removed and their project page deleted, so nothing inherited from it remains.This repository is not affiliated with, endorsed by, or released by the original authors, UC Berkeley, or the Berkeley Speech Group.
One narrow exception, kept deliberately separate: analysis tooling may contain a clearly-marked port of the reference implementation, used only as a fixed reference to A/B against. Any such file names its source in its module docstring, carries the original license notice, and is not part of the reimplementation itself.
This repository ships no audio samples, figures, or demo pages — the ones it used to carry were the original authors' and have been removed. There is nothing to demo yet: the models here have not been trained. For the authors' own samples, see their project page linked above.
Overview
StyleStream is a real-time zero-shot voice style conversion system that transforms the timbre, accent, and emotion of speech without any fine-tuning on the target speaker or style. It achieves state-of-the-art conversion quality with an end-to-end streaming latency of approximately 1 second, using a three-stage pipeline: content extraction (Destylizer), style-conditioned synthesis (Stylizer), and waveform generation (Vocoder). All components operate at a unified 50 Hz frame rate on 16 kHz audio.
Architecture
Style Reference
|
v
[Style Encoder]
(WavLM-TDNN)
|
v
Source Audio ---> [Destylizer] ---> Content ---> [Stylizer] ---> Mel ---> [Vocoder] ---> Converted Audio
HuBERT L18 Features DiT x16 Spectrogram Causal (16 kHz)
Conformer x6 (50 Hz) CFM + CFG (100 bins) Vocos
FSQ [5,3,3] adaLN-Zero ConvNeXt x8
ISTFT
Destylizer -- Extracts style-invariant content features. HuBERT-Large layer 18 feeds into 6 Conformer blocks with ALiBi positional encoding, quantized through FSQ with codebook size 45. Trained with CTC + seq2seq ASR losses.
Stylizer -- Generates mel spectrograms conditioned on content and style. 16-layer Diffusion Transformer with RoPE, Conditional Flow Matching (OT path + Euler sampling), adaLN-Zero conditioning from a WavLM-TDNN style encoder, and Classifier-Free Guidance (alpha=2).
Vocoder -- Converts mel spectrograms to waveforms. Causal Vocos architecture with 8 ConvNeXt V2 blocks using causal depthwise separable convolutions, ISTFT head for waveform synthesis, and GAN training with multi-scale discriminator.
Features
- Zero-shot conversion -- No fine-tuning required for new speakers or styles
- Real-time streaming -- End-to-end latency of ~1 second using chunked causal attention with 600ms chunks
- Multi-style transfer -- Supports timbre, accent, and emotion conversion
- Streaming-optimized -- KV caching, StreamingHuBERT, ring buffer pipeline, and MSE distillation for efficient inference
- Training optimizations -- Flash Attention (SDPA), torch.compile, Lion optimizer, Grouped Query Attention (GQA), gradient checkpointing, progressive training, Min-SNR loss weighting
- Fast experimentation -- Reduced-size configs (
configs/*/fast.yaml) for 2-3x faster training;--micro Nfor rapid prototyping on small subsets - Preprocessing acceleration -- Merged resample+mel pipeline (80% I/O reduction), GPU-batched HuBERT extraction with duration sorting, FP16 mel/feature storage, pipeline parallelism
Project Structure
stylestream/
config.py # Structured configuration dataclasses
destylizer/ # ALiBi, Conformer x6, FSQ, ASR decoder, trainer
stylizer/ # RoPE, DiT x16, CFM, adaLN-Zero, style encoder, CFG, trainer
vocoder/ # Causal ConvNeXt, ISTFT head, discriminator, GAN trainer
streaming/ # Chunked attention, KV cache, StreamingHuBERT, distillation, pipeline
data/ # Manifests, preprocessing, HuBERT extraction, datasets
eval/ # Whisper WER, Resemblyzer S-SIM, ECAPA A-SIM, emotion2vec E-SIM, UTMOS
training/ # Base trainer, scheduler, distributed training
utils/ # Mel, audio, logging, checkpointing utilities
configs/ # YAML configs (destylizer, stylizer, vocoder, streaming, eval, + fast.yaml variants)
scripts/ # CLI entry points for training, inference, evaluation
tests/ # 568 tests across all modules
Installation
Requires Python 3.12+. Uses uv for package management.
git clone https://github.com/ayutaz/StyleStream.git
cd StyleStream
# Core dependencies only
uv sync
# Full installation (training + evaluation + development)
uv sync --extra train --extra eval --extra dev
Dependencies
Core: torch, torchaudio, transformers, accelerate, einops, hydra-core, omegaconf
Training: wandb, tensorboard, datasets
Evaluation: resemblyzer, jiwer, matplotlib
Quick Start
Data Preprocessing
# Download datasets
uv run python scripts/download_libritts.py --output-dir data/raw/libritts
uv run python scripts/download_esd.py --output-dir data/raw/esd
# Download pretrained feature extractors
uv run python scripts/download_models.py --stage train
# Preprocess (merged resample + mel computation + HuBERT feature extraction)
uv run python scripts/preprocess_data.py --manifest data/manifests/libritts.csv --output-dir data/processed
# Preprocess with pipeline parallelism (overlap resample and mel stages)
uv run python scripts/preprocess_data.py --manifest data/manifests/libritts.csv --output-dir data/processed --pipelined
# Micro-dataset for rapid prototyping (stratified sample of N utterances)
uv run python scripts/preprocess_data.py --manifest data/manifests/libritts.csv --output-dir data/processed --micro 1000
# Validate extracted features
uv run python scripts/validate_features.py --manifest data/manifests/libritts.csv --processed-dir data/processed
Training
# Stage 1: Destylizer (Conformer + FSQ with ASR loss)
uv run python scripts/train_destylizer.py --config configs/destylizer/offline.yaml
# Stage 2: Stylizer (DiT + CFM with spectral inpainting)
uv run python scripts/train_stylizer.py --config configs/stylizer/offline.yaml
# Stage 3: Vocoder (Causal Vocos with GAN training)
uv run python scripts/train_vocoder.py --config configs/vocoder/causal_vocos.yaml
# Stage 4: Streaming adaptation (MSE distillation + fine-tuning)
uv run python scripts/train_streaming_destylizer.py --config configs/streaming/distillation.yaml
uv run python scripts/train_streaming_stylizer.py --config configs/streaming/stylizer.yaml
Fast Training
Fast configs (configs/*/fast.yaml) use reduced model sizes and optimizations for 2-3x faster experimentation at ~85-90% of full quality. Changes include fewer layers, smaller FFN, Lion optimizer, torch.compile, and progressive training.
# Fast Destylizer (Conformer x4, FFN 2048, 50k steps)
uv run python scripts/train_destylizer.py --config configs/destylizer/fast.yaml
# Fast Stylizer (DiT x10, FFN 2048, Lion optimizer, progressive training, 200k steps)
uv run python scripts/train_stylizer.py --config configs/stylizer/fast.yaml
# Fast Vocoder (intermediate 1024, 50k steps)
uv run python scripts/train_vocoder.py --config configs/vocoder/fast.yaml
Inference
# Offline (full-utterance) conversion
uv run python scripts/inference.py \
--source source.wav --reference target_style.wav -o converted.wav
# Streaming conversion (~1s latency)
uv run python scripts/inference.py \
--source source.wav --reference target_style.wav --streaming
# Streaming inference demo with ring buffer pipeline
uv run python scripts/streaming_inference.py \
--source source.wav --target target_style.wav --output converted.wav
# Batch conversion from evaluation pairs
uv run python scripts/inference.py \
--batch pairs.csv --output-dir converted/
Evaluation
# Run full evaluation (WER, S-SIM, A-SIM, E-SIM, UTMOS)
uv run python scripts/evaluate.py \
--converted-dir eval_results/converted --pairs pairs.csv
# Evaluate specific metrics only
uv run python scripts/evaluate.py \
--converted-dir eval_results/converted --pairs pairs.csv \
--metrics wer,s_sim
# With paper baselines for comparison
uv run python scripts/evaluate.py \
--converted-dir eval_results/converted --pairs pairs.csv \
--config configs/eval/stylestream_test.yaml --output-dir eval_results
Testing
# Run all 568 tests
uv run pytest tests/ -v
# Run tests for a specific module
uv run pytest tests/test_conformer.py -v
uv run pytest tests/test_cfm.py -v
uv run pytest tests/test_streaming_models.py -v
Optimizations
Training
| Optimization | Description |
|---|---|
| Flash Attention (SDPA) | F.scaled_dot_product_attention in Conformer and DiT for automatic Flash/memory-efficient kernel dispatch |
torch.compile | Optional compilation with reduce-overhead mode (enabled in fast configs via compile_model: true) |
| Lion optimizer | Momentum-only optimizer with 2x memory efficiency vs AdamW (Chen et al., 2023) |
| Grouped Query Attention | GQA in DiT blocks -- configurable num_kv_heads reduces KV memory while preserving quality |
| Gradient checkpointing | Optional per-block checkpointing in DiT to reduce activation memory |
| Progressive training | 3-stage curriculum: gradually increases segment length (3s -> 4.5s -> 6s) and mask ratio |
| Style embedding cache | Pre-computed style embeddings to skip WavLM forward pass during Stylizer training |
| ALiBi/RoPE caching | Positional encoding tensors cached and reused across forward passes |
| Mixed precision (bf16) | All components trained in bf16; mel spectrograms and HuBERT features stored as float16 |
| CUDA optimizations | cudnn.benchmark, TF32 matmul, set_float32_matmul_precision("high") enabled by default |
Preprocessing
| Optimization | Description |
|---|---|
| Merged resample+mel | Single-pass pipeline eliminates intermediate WAV I/O (~80% I/O reduction) |
| GPU-batched HuBERT | Duration-sorted batching (batch_size=16) minimizes padding waste on GPU |
| FP16 storage | Mel spectrograms and HuBERT features stored as float16 (halves disk usage and I/O) |
| Pipeline parallelism | --pipelined flag overlaps resample and mel stages across chunks |
| Micro-dataset | --micro N creates stratified subsets for rapid architecture validation |
| Manifest API | sort_by_duration, shard, filter_valid, stratified_sample for flexible data management |
Fast Configs
| Config | Key Changes | Steps |
|---|---|---|
configs/destylizer/fast.yaml | Conformer x4, FFN 2048, ASR decoder x2 | 50k (vs 80k) |
configs/stylizer/fast.yaml | DiT x10, FFN 2048, Lion, torch.compile, progressive | 200k (vs 320k) |
configs/vocoder/fast.yaml | intermediate_size 1024 | 50k (vs 80k) |
Implementation Status
| Phase | Component | Description | Status |
|---|---|---|---|
| P0 | Infrastructure | Config dataclasses, training base, utilities, checkpoint management | Done |
| P1 | Data Pipeline | Manifests (LibriTTS/ESD/GLOBE), mel preprocessing, HuBERT extraction, datasets | Done |
| P2 | Destylizer | Conformer x6 with ALiBi, FSQ [5,3,3], CTC + seq2seq ASR decoder | Done |
| P3 | Stylizer | 16-layer DiT, CFM (OT path), adaLN-Zero, WavLM-TDNN style encoder, CFG | Done |
| P4 | Vocoder | Causal Vocos (ConvNeXt x8, ISTFT), multi-scale discriminator, GAN training | Done |
| P5 | Streaming | Chunked causal attention, KV cache, StreamingHuBERT, MSE distillation, ring buffer | Done |
| P6 | Evaluation | Whisper WER/CER, Resemblyzer S-SIM, ECAPA A-SIM, emotion2vec E-SIM, UTMOS, visualization | Done |
568 tests covering all modules -- passing.
Paper Target Metrics
Reference baselines from Table 1 of the paper (StyleStream-Test, 3000 pairs):
| Mode | WER (%) | S-SIM | A-SIM | E-SIM |
|---|---|---|---|---|
| Ground Truth | 3.8 | -- | -- | -- |
| Offline | 9.2 | 0.852 | 0.640 | 0.827 |
| Streaming | 10.7 | 0.837 | 0.626 | 0.733 |
- WER: Word Error Rate via Whisper-large-v3 (lower is better)
- S-SIM: Speaker/timbre similarity via Resemblyzer (higher is better)
- A-SIM: Accent similarity via ECAPA-TDNN accent-ID (higher is better)
- E-SIM: Emotion similarity via emotion2vec (higher is better)
Citation
The method is not mine. If this repository is useful to you, cite the paper it reimplements:
@article{liu2026stylestream,
title={StyleStream: Real-Time Zero-Shot Voice Style Conversion},
author={Yisi Liu, Nicholas Lee, Gopala Anumanchipalli},
journal={arXiv preprint arXiv:2602.20113},
year={2026}
}
License
The code in this repository is released under the MIT License, Copyright (c) 2026 ayutaz. It is an independent reimplementation from the paper, so the original authors' code license does not apply to it (see Provenance above).
The MIT license covers this source code only. It does not, and cannot, grant rights to the things this code depends on:
| What | Whose terms apply |
|---|---|
| Pretrained weights (WavLM, HuBERT, Vocos, Whisper, evaluation models) | Each publisher's own license |
| Training/evaluation datasets | Emilia is CC BY-NC (non-commercial); MSP-Podcast needs a signed academic agreement; LibriTTS is CC BY 4.0 |
| Weights you produce by training this code | Constrained by the licenses of the data used |
Artifacts released by the original authors (checkpoints, target_spkrs.tar, reference code) | UC Regents Research, Educational, and Not-for-Profit License |
So "MIT" does not mean the trained system is free for any use. In particular, a model trained on Emilia inherits a non-commercial constraint from the data regardless of this license. Check the upstream terms for your intended use.
Contributors
Similar repos
Languages
Python
100.0%
README
StyleStream
A complete PyTorch reimplementation of StyleStream: Real-Time Zero-Shot Voice Style Conversion (arXiv:2602.20113).
Provenance
This is an independent, from-scratch reimplementation written from the published paper alone. No code from the original authors was used, viewed, or adapted while building it. Every module here was derived by reading the paper and reconstructing the described architecture in PyTorch.
The timeline is verifiable in this repository's git history: the style encoder landed 2026-03-16, roughly three months before the original authors published their model code (2026-06-11). Where this implementation differs from theirs, the difference is an interpretation gap, not a deviation from a reference that was available at the time.
The history explains itself if you read it. The first few commits are authored by the original authors, because this repository began in February 2026 as a fork of theirs — back when that repository held nothing but a project page: a
LICENSE, aREADME.md, anddocs/index.html. Not one.pyfile. There was no model code to copy, because none had been published. Everything understylestream/was written afterwards, from the paper. The fork link has since been removed and their project page deleted, so nothing inherited from it remains.This repository is not affiliated with, endorsed by, or released by the original authors, UC Berkeley, or the Berkeley Speech Group.
One narrow exception, kept deliberately separate: analysis tooling may contain a clearly-marked port of the reference implementation, used only as a fixed reference to A/B against. Any such file names its source in its module docstring, carries the original license notice, and is not part of the reimplementation itself.
This repository ships no audio samples, figures, or demo pages — the ones it used to carry were the original authors' and have been removed. There is nothing to demo yet: the models here have not been trained. For the authors' own samples, see their project page linked above.
Overview
StyleStream is a real-time zero-shot voice style conversion system that transforms the timbre, accent, and emotion of speech without any fine-tuning on the target speaker or style. It achieves state-of-the-art conversion quality with an end-to-end streaming latency of approximately 1 second, using a three-stage pipeline: content extraction (Destylizer), style-conditioned synthesis (Stylizer), and waveform generation (Vocoder). All components operate at a unified 50 Hz frame rate on 16 kHz audio.
Architecture
Style Reference
|
v
[Style Encoder]
(WavLM-TDNN)
|
v
Source Audio ---> [Destylizer] ---> Content ---> [Stylizer] ---> Mel ---> [Vocoder] ---> Converted Audio
HuBERT L18 Features DiT x16 Spectrogram Causal (16 kHz)
Conformer x6 (50 Hz) CFM + CFG (100 bins) Vocos
FSQ [5,3,3] adaLN-Zero ConvNeXt x8
ISTFT
Destylizer -- Extracts style-invariant content features. HuBERT-Large layer 18 feeds into 6 Conformer blocks with ALiBi positional encoding, quantized through FSQ with codebook size 45. Trained with CTC + seq2seq ASR losses.
Stylizer -- Generates mel spectrograms conditioned on content and style. 16-layer Diffusion Transformer with RoPE, Conditional Flow Matching (OT path + Euler sampling), adaLN-Zero conditioning from a WavLM-TDNN style encoder, and Classifier-Free Guidance (alpha=2).
Vocoder -- Converts mel spectrograms to waveforms. Causal Vocos architecture with 8 ConvNeXt V2 blocks using causal depthwise separable convolutions, ISTFT head for waveform synthesis, and GAN training with multi-scale discriminator.
Features
- Zero-shot conversion -- No fine-tuning required for new speakers or styles
- Real-time streaming -- End-to-end latency of ~1 second using chunked causal attention with 600ms chunks
- Multi-style transfer -- Supports timbre, accent, and emotion conversion
- Streaming-optimized -- KV caching, StreamingHuBERT, ring buffer pipeline, and MSE distillation for efficient inference
- Training optimizations -- Flash Attention (SDPA), torch.compile, Lion optimizer, Grouped Query Attention (GQA), gradient checkpointing, progressive training, Min-SNR loss weighting
- Fast experimentation -- Reduced-size configs (
configs/*/fast.yaml) for 2-3x faster training;--micro Nfor rapid prototyping on small subsets - Preprocessing acceleration -- Merged resample+mel pipeline (80% I/O reduction), GPU-batched HuBERT extraction with duration sorting, FP16 mel/feature storage, pipeline parallelism
Project Structure
stylestream/
config.py # Structured configuration dataclasses
destylizer/ # ALiBi, Conformer x6, FSQ, ASR decoder, trainer
stylizer/ # RoPE, DiT x16, CFM, adaLN-Zero, style encoder, CFG, trainer
vocoder/ # Causal ConvNeXt, ISTFT head, discriminator, GAN trainer
streaming/ # Chunked attention, KV cache, StreamingHuBERT, distillation, pipeline
data/ # Manifests, preprocessing, HuBERT extraction, datasets
eval/ # Whisper WER, Resemblyzer S-SIM, ECAPA A-SIM, emotion2vec E-SIM, UTMOS
training/ # Base trainer, scheduler, distributed training
utils/ # Mel, audio, logging, checkpointing utilities
configs/ # YAML configs (destylizer, stylizer, vocoder, streaming, eval, + fast.yaml variants)
scripts/ # CLI entry points for training, inference, evaluation
tests/ # 568 tests across all modules
Installation
Requires Python 3.12+. Uses uv for package management.
git clone https://github.com/ayutaz/StyleStream.git
cd StyleStream
# Core dependencies only
uv sync
# Full installation (training + evaluation + development)
uv sync --extra train --extra eval --extra dev
Dependencies
Core: torch, torchaudio, transformers, accelerate, einops, hydra-core, omegaconf
Training: wandb, tensorboard, datasets
Evaluation: resemblyzer, jiwer, matplotlib
Quick Start
Data Preprocessing
# Download datasets
uv run python scripts/download_libritts.py --output-dir data/raw/libritts
uv run python scripts/download_esd.py --output-dir data/raw/esd
# Download pretrained feature extractors
uv run python scripts/download_models.py --stage train
# Preprocess (merged resample + mel computation + HuBERT feature extraction)
uv run python scripts/preprocess_data.py --manifest data/manifests/libritts.csv --output-dir data/processed
# Preprocess with pipeline parallelism (overlap resample and mel stages)
uv run python scripts/preprocess_data.py --manifest data/manifests/libritts.csv --output-dir data/processed --pipelined
# Micro-dataset for rapid prototyping (stratified sample of N utterances)
uv run python scripts/preprocess_data.py --manifest data/manifests/libritts.csv --output-dir data/processed --micro 1000
# Validate extracted features
uv run python scripts/validate_features.py --manifest data/manifests/libritts.csv --processed-dir data/processed
Training
# Stage 1: Destylizer (Conformer + FSQ with ASR loss)
uv run python scripts/train_destylizer.py --config configs/destylizer/offline.yaml
# Stage 2: Stylizer (DiT + CFM with spectral inpainting)
uv run python scripts/train_stylizer.py --config configs/stylizer/offline.yaml
# Stage 3: Vocoder (Causal Vocos with GAN training)
uv run python scripts/train_vocoder.py --config configs/vocoder/causal_vocos.yaml
# Stage 4: Streaming adaptation (MSE distillation + fine-tuning)
uv run python scripts/train_streaming_destylizer.py --config configs/streaming/distillation.yaml
uv run python scripts/train_streaming_stylizer.py --config configs/streaming/stylizer.yaml
Fast Training
Fast configs (configs/*/fast.yaml) use reduced model sizes and optimizations for 2-3x faster experimentation at ~85-90% of full quality. Changes include fewer layers, smaller FFN, Lion optimizer, torch.compile, and progressive training.
# Fast Destylizer (Conformer x4, FFN 2048, 50k steps)
uv run python scripts/train_destylizer.py --config configs/destylizer/fast.yaml
# Fast Stylizer (DiT x10, FFN 2048, Lion optimizer, progressive training, 200k steps)
uv run python scripts/train_stylizer.py --config configs/stylizer/fast.yaml
# Fast Vocoder (intermediate 1024, 50k steps)
uv run python scripts/train_vocoder.py --config configs/vocoder/fast.yaml
Inference
# Offline (full-utterance) conversion
uv run python scripts/inference.py \
--source source.wav --reference target_style.wav -o converted.wav
# Streaming conversion (~1s latency)
uv run python scripts/inference.py \
--source source.wav --reference target_style.wav --streaming
# Streaming inference demo with ring buffer pipeline
uv run python scripts/streaming_inference.py \
--source source.wav --target target_style.wav --output converted.wav
# Batch conversion from evaluation pairs
uv run python scripts/inference.py \
--batch pairs.csv --output-dir converted/
Evaluation
# Run full evaluation (WER, S-SIM, A-SIM, E-SIM, UTMOS)
uv run python scripts/evaluate.py \
--converted-dir eval_results/converted --pairs pairs.csv
# Evaluate specific metrics only
uv run python scripts/evaluate.py \
--converted-dir eval_results/converted --pairs pairs.csv \
--metrics wer,s_sim
# With paper baselines for comparison
uv run python scripts/evaluate.py \
--converted-dir eval_results/converted --pairs pairs.csv \
--config configs/eval/stylestream_test.yaml --output-dir eval_results
Testing
# Run all 568 tests
uv run pytest tests/ -v
# Run tests for a specific module
uv run pytest tests/test_conformer.py -v
uv run pytest tests/test_cfm.py -v
uv run pytest tests/test_streaming_models.py -v
Optimizations
Training
| Optimization | Description |
|---|---|
| Flash Attention (SDPA) | F.scaled_dot_product_attention in Conformer and DiT for automatic Flash/memory-efficient kernel dispatch |
torch.compile | Optional compilation with reduce-overhead mode (enabled in fast configs via compile_model: true) |
| Lion optimizer | Momentum-only optimizer with 2x memory efficiency vs AdamW (Chen et al., 2023) |
| Grouped Query Attention | GQA in DiT blocks -- configurable num_kv_heads reduces KV memory while preserving quality |
| Gradient checkpointing | Optional per-block checkpointing in DiT to reduce activation memory |
| Progressive training | 3-stage curriculum: gradually increases segment length (3s -> 4.5s -> 6s) and mask ratio |
| Style embedding cache | Pre-computed style embeddings to skip WavLM forward pass during Stylizer training |
| ALiBi/RoPE caching | Positional encoding tensors cached and reused across forward passes |
| Mixed precision (bf16) | All components trained in bf16; mel spectrograms and HuBERT features stored as float16 |
| CUDA optimizations | cudnn.benchmark, TF32 matmul, set_float32_matmul_precision("high") enabled by default |
Preprocessing
| Optimization | Description |
|---|---|
| Merged resample+mel | Single-pass pipeline eliminates intermediate WAV I/O (~80% I/O reduction) |
| GPU-batched HuBERT | Duration-sorted batching (batch_size=16) minimizes padding waste on GPU |
| FP16 storage | Mel spectrograms and HuBERT features stored as float16 (halves disk usage and I/O) |
| Pipeline parallelism | --pipelined flag overlaps resample and mel stages across chunks |
| Micro-dataset | --micro N creates stratified subsets for rapid architecture validation |
| Manifest API | sort_by_duration, shard, filter_valid, stratified_sample for flexible data management |
Fast Configs
| Config | Key Changes | Steps |
|---|---|---|
configs/destylizer/fast.yaml | Conformer x4, FFN 2048, ASR decoder x2 | 50k (vs 80k) |
configs/stylizer/fast.yaml | DiT x10, FFN 2048, Lion, torch.compile, progressive | 200k (vs 320k) |
configs/vocoder/fast.yaml | intermediate_size 1024 | 50k (vs 80k) |
Implementation Status
| Phase | Component | Description | Status |
|---|---|---|---|
| P0 | Infrastructure | Config dataclasses, training base, utilities, checkpoint management | Done |
| P1 | Data Pipeline | Manifests (LibriTTS/ESD/GLOBE), mel preprocessing, HuBERT extraction, datasets | Done |
| P2 | Destylizer | Conformer x6 with ALiBi, FSQ [5,3,3], CTC + seq2seq ASR decoder | Done |
| P3 | Stylizer | 16-layer DiT, CFM (OT path), adaLN-Zero, WavLM-TDNN style encoder, CFG | Done |
| P4 | Vocoder | Causal Vocos (ConvNeXt x8, ISTFT), multi-scale discriminator, GAN training | Done |
| P5 | Streaming | Chunked causal attention, KV cache, StreamingHuBERT, MSE distillation, ring buffer | Done |
| P6 | Evaluation | Whisper WER/CER, Resemblyzer S-SIM, ECAPA A-SIM, emotion2vec E-SIM, UTMOS, visualization | Done |
568 tests covering all modules -- passing.
Paper Target Metrics
Reference baselines from Table 1 of the paper (StyleStream-Test, 3000 pairs):
| Mode | WER (%) | S-SIM | A-SIM | E-SIM |
|---|---|---|---|---|
| Ground Truth | 3.8 | -- | -- | -- |
| Offline | 9.2 | 0.852 | 0.640 | 0.827 |
| Streaming | 10.7 | 0.837 | 0.626 | 0.733 |
- WER: Word Error Rate via Whisper-large-v3 (lower is better)
- S-SIM: Speaker/timbre similarity via Resemblyzer (higher is better)
- A-SIM: Accent similarity via ECAPA-TDNN accent-ID (higher is better)
- E-SIM: Emotion similarity via emotion2vec (higher is better)
Citation
The method is not mine. If this repository is useful to you, cite the paper it reimplements:
@article{liu2026stylestream,
title={StyleStream: Real-Time Zero-Shot Voice Style Conversion},
author={Yisi Liu, Nicholas Lee, Gopala Anumanchipalli},
journal={arXiv preprint arXiv:2602.20113},
year={2026}
}
License
The code in this repository is released under the MIT License, Copyright (c) 2026 ayutaz. It is an independent reimplementation from the paper, so the original authors' code license does not apply to it (see Provenance above).
The MIT license covers this source code only. It does not, and cannot, grant rights to the things this code depends on:
| What | Whose terms apply |
|---|---|
| Pretrained weights (WavLM, HuBERT, Vocos, Whisper, evaluation models) | Each publisher's own license |
| Training/evaluation datasets | Emilia is CC BY-NC (non-commercial); MSP-Podcast needs a signed academic agreement; LibriTTS is CC BY 4.0 |
| Weights you produce by training this code | Constrained by the licenses of the data used |
Artifacts released by the original authors (checkpoints, target_spkrs.tar, reference code) | UC Regents Research, Educational, and Not-for-Profit License |
So "MIT" does not mean the trained system is free for any use. In particular, a model trained on Emilia inherits a non-commercial constraint from the data regardless of this license. Check the upstream terms for your intended use.
Similar repos
Contributors
Languages
Python
100.0%