DSA-Tokenizer: Disentangled Semantic-Acoustic Tokenization via Flow Matching-based Hierarchical Fusion

Installation

# Create the same Python major/minor version used by this repo
conda create -n DSA-Tokenizer python=3.11.15
conda activate DSA-Tokenizer

# Clone and install the environment used by this repo
git clone https://github.com/discrete-speech/DSA-Tokenizer.git
cd DSA-Tokenizer
pip install -e .

FlashAttention is optional for inference. DSATokenizer.from_pretrained(...) disables FlashAttention and can run without installing flash-attn.

Training configs use model.arch.use_flash_attn=true by default. Install FlashAttention only if you plan to train or fine-tune:

bash scripts/install_flash_attn.sh

Inference with the DSA-Tokenizer API

from pathlib import Path

from f5_tts.dsa_api import DSATokenizer

example_dir = Path("examples/dsa_seedtts")
output_dir = Path("dsa_seedtts_api_demo")
output_dir.mkdir(parents=True, exist_ok=True)

target_audio = example_dir / "target.wav"
prompt_audio = example_dir / "prompt.wav"
target_semantic = example_dir / "target.semantic.txt"
prompt_semantic = example_dir / "prompt.semantic.txt"

dsa = DSATokenizer.from_pretrained("DiscreteSpeech/dsa-tokenizer", device="cuda:0")

# 1. Extract acoustic tokens from an audio file.
acoustic_info = dsa.extract_acoustic(
    audio_file=str(target_audio),
    semantic_tokens_file=str(target_semantic),
    file_json=str(output_dir / "target_acoustic_tokens.json"),
    file_acoustic_tokens=str(output_dir / "target.acoustic.txt"),
)

# 2. Reconstruct the target audio from its semantic tokens and acoustic condition.
wav, sr, spec = dsa.reconstruct(
    audio_file=str(target_audio),
    semantic_tokens_file=str(target_semantic),
    file_wave=str(output_dir / "target_reconstruction.wav"),
)

# 3. Voice cloning: use prompt.wav as the acoustic/speaker reference,
#    and synthesize speech following target.semantic.txt.
wav, sr, spec = dsa.voice_clone(
    ref_file=str(prompt_audio),
    ref_semantic_tokens_file=str(prompt_semantic),
    gen_semantic_tokens_file=str(target_semantic),
    file_wave=str(output_dir / "target_voice_cloned_from_prompt.wav"),
)

The semantic tokenizer is released in the DSTK project. The semantic tokens were extracted using code:

import sys
import librosa

dstk_path = "/path/to/DSTK"
sys.path.append(dstk_path)

input_wav_path = dstk_path + "/004892.wav"

from semantic_tokenizer.f40ms.simple_tokenizer_infer import SpeechTokenizer

tokenizer = SpeechTokenizer()

raw_wav, sr = librosa.load(input_wav_path, sr=16000)
token_list, token_info_list = tokenizer.extract([raw_wav])  
for token_info in token_info_list:
    print(token_info["unit_sequence"] + "\n")
    print(token_info["reduced_unit_sequence"] + "\n")

You can also pass the whitespace-separated token string directly via semantic_tokens, ref_semantic_tokens, or gen_semantic_tokens instead of using *_semantic_tokens_file.

Training

Prepare Dataset

1. Download the Chinese and English subsets of Emilia from Hugging Face, untar all audio files, and extract WavLM embeddings for each audio file. For WavLM embedding extraction, refer to: https://github.com/microsoft/UniSpeech/tree/main/downstreams/speaker_verification. For each audio file, the corresponding WavLM embedding should be stored in the same directory with the same basename and a .pt extension. The data structure should look like this:

    ├── EN/
    │   ├── EN-B000000/
    │   │   ├── xxxxx.wav
    │   │   ├── xxxxx.pt
    │   │   └── xxxxx.json
    │   ├── EN-B000001/
    │   │   ├── xxxxx.wav
    │   │   ├── xxxxx.pt
    │   │   └── xxxxx.json
    │   └── ...
    └── ZH/
        ├── ZH-B000000/
        │   ├── xxxxx.wav
        │   ├── xxxxx.pt
        │   └── xxxxx.json
        ├── ZH-B000001/
        │   ├── xxxxx.wav
        │   ├── xxxxx.pt
        │   └── xxxxx.json
        └── ...
   

2. Download the pre-extracted semantic tokens for Emilia from Hugging Face:

   hf download DiscreteSpeech/emilia_semantic_token \
       emilia_semantic_token.tar.gz \
       --repo-type dataset \
       --local-dir PATH/TO/DEST
   

   After downloading, extract the archive. Each extracted file contains semantic token sequences for one Emilia subset. Each line follows the format:

    {audio path}|{semantic token sequence}|{text} # Replace {audio path} with the corresponding local path to the audio file.
   

3. Build the training dataset

   We use pyannote/speaker-diarization-3.1 to filter out audio samples that contain more than one speaker.

   You can download the cleaned file lists from Hugging Face:

   hf download DiscreteSpeech/emilia_cleaning_result cleaning_results_en.zip \
       --repo-type dataset \
       --local-dir PATH/TO/DEST
   
   hf download DiscreteSpeech/emilia_cleaning_result cleaning_results_zh.zip \
       --repo-type dataset \
       --local-dir PATH/TO/DEST
   

   Then unzip the downloaded files, after unzip, replace {audio path} in the .txt files with the corresponding local path to your audio files.

   Finally, set input_path, dirty_en_path, dirty_zh_path, and save_dir in src/f5_tts/train/datasets/my_prepare_Emilia_wav_unit_text_clean.py, then run:

   python src/f5_tts/train/datasets/my_prepare_Emilia_wav_unit_text_clean.py
   

Configure Accelerate

Training uses accelerate launch, but this repository does not ship a fixed Accelerate config because the correct values depend on your machine. Before running any training command, create a local Accelerate config once:

accelerate config

For a single-node multi-GPU run, choose the multi-GPU option, set the number of processes to the number of GPUs you want to use, and select bf16 or fp16 mixed precision according to your GPU support. Accelerate saves this machine local config under your user cache directory, for example ~/.cache/huggingface/accelerate/default_config.yaml.

For a simple single-GPU smoke test, the default config is also enough:

accelerate config default

Training DSA-Tokenizer

Set the prepared dataset path in each training command. By default, checkpoints are saved to ckpts/${model.name}_${model.mel_spec.mel_spec_type}_${model.tokenizer} and then used by the next stage:

DATASET_PATH=/path/to/Emilia_EN_ZH_clean_unit_4096

# produced by acoustic token learning
FLOW_MATCHING_CKPT=ckpts/train_dsa_tokenizer_flow_matching_flow2gan_unit_4096/model_400000.pt

# produced by distillation stage 1
DISTILL_STAGE1_CKPT=ckpts/distill_dsa_tokenizer_stage_1_flow2gan_unit_4096/model_10000.pt

# produced by distillation stage 2
DISTILL_STAGE2_CKPT=ckpts/distill_dsa_tokenizer_stage_2_flow2gan_unit_4096/model_10000.pt

For Acoustic Token learning part:

## For acoustic token learning
## Train this flow matching stage for 400k steps.
accelerate launch src/f5_tts/train/train.py \
    --config-name train_dsa_tokenizer_flow_matching.yaml \
    datasets.path=${DATASET_PATH}

For Distill stage1 of DSA-Tokenizer:

## For DSA-Tokenizer distillation stage 1
## Use the 400k-step checkpoint from flow matching training as teacher_ckpt.
## Train this distillation stage for 10k steps.
accelerate launch src/f5_tts/train/train_distill.py \
    --config-name distill_dsa_tokenizer_stage_1.yaml \
    datasets.path=${DATASET_PATH} \
    distill.teacher_ckpt=${FLOW_MATCHING_CKPT}

For Distill stage2 of DSA-Tokenizer:

## For DSA-Tokenizer distillation stage 2
## Use the 10k-step checkpoint from distillation stage 1 as teacher_ckpt.
## Train this distillation stage for another 10k steps.
accelerate launch src/f5_tts/train/train_distill.py \
    --config-name distill_dsa_tokenizer_stage_2.yaml \
    datasets.path=${DATASET_PATH} \
    distill.teacher_ckpt=${DISTILL_STAGE1_CKPT}

For GAN-Finetuning part:

Download our pretrained proxy mel speaker encoder:

hf download DiscreteSpeech/proxy_mel_speaker_encoder \
    proxy_mel_speaker_encoder.pt \
    --repo-type model \
    --local-dir ckpts/pretrained/mel_speaker_encoder

## For GAN-Finetuning part
## Use the 10k-step checkpoint from distillation stage 2 as init_checkpoint.
## Train GAN finetuning for another 15k steps.
accelerate launch src/f5_tts/train/train_gan_finetune.py \
    --config-name gan_fine_tuning.yaml \
    datasets.path=${DATASET_PATH} \
    train.init_checkpoint=${DISTILL_STAGE2_CKPT}

Citation

If you find our work useful, please consider citing:

@misc{zhang2026dsatokenizerdisentangledsemanticacoustictokenization,
      title={DSA-Tokenizer: Disentangled Semantic-Acoustic Tokenization via Flow Matching-based Hierarchical Fusion},
      author={Hanlin Zhang and Daxin Tan and Dehua Tao and Xiao Chen and Haochen Tan and Yunhe Li and Yuchen Cao and Linqi Song},
      year={2026},
      eprint={2601.09239},
      archivePrefix={arXiv},
      primaryClass={cs.SD},
      url={https://arxiv.org/abs/2601.09239},
}

Acknowledgements

We sincerely thank the authors and maintainers of F5-TTS and DualCodec for their open-source codebases. DSA-Tokenizer benefits from their excellent implementations and research infrastructure for flow matching-based speech generation and neural audio codec modeling.