VoiceStudio

A unified toolkit for voice cloning, designing and editing.

---

🎯 Overview

Speech synthesis research is held back by its own tooling. Every model arrives as its own repository, with its own runtime, its own checkpoint format, its own inference script and its own pinned dependency set, and most of them ship weights you can run but no path to training them. Comparing two models means learning two codebases; building on one means adopting it wholesale.

VoiceStudio removes that tax. Every model here is an ordinary transformers model: a PreTrainedConfig, a PreTrainedModel and a Processor, loaded with from_pretrained, run with generate, trained with forward(labels=...). Swapping one for another is changing a class name. Comparing them is a loop. Fine tuning one is the training code you already have.

Key Features:

• One API: every model takes its inputs from its own Processor and returns audio the same processor decodes, so switching models is switching a class name
• Composable: models hold each other as ordinary submodels, so Parler-TTS owns a DacModel, Chroma a MimiModel, and F5-TTS whichever of VocosModel or BigVGANModel its checkpoint was trained against
• Inheritance over reimplementation: rebased onto llama, qwen3, csm, mimi, dac, speecht5 and a dozen more, and onto each other, rather than carrying parallel copies
• Trainable, not inference only: every model returns a loss, with the objective read out of the upstream project's own trainer rather than guessed from its shape
• Verified against published weights: loaded from the real checkpoint, made to speak, and the audio transcribed back and compared to the text it was given
• Direct loading: from_pretrained on the official repository id, with no conversion step for the caller to run
• Fewer dependencies: a migration ends an upstream import rather than adding one, leaving transformers as the only required dependency and everything else behind an extra

---

🛠️ Installation

Python 3.13 or newer, and PyTorch 2.8 or newer.

The base install carries only transformers[kernels]. The runtime and the audio stack are extras, so pick the ones for the machine you are on.

From source

git clone https://github.com/LatentForge/VoiceStudio.git
cd VoiceStudio
uv sync --extra cloud --extra audio

For research

git clone https://github.com/LatentForge/VoiceStudio.git
cd VoiceStudio
uv sync --extra research

Use uv sync rather than pip install. torch is pinned to a specific index for Windows in [tool.uv.sources], and pip ignores that file. The voicestudio distribution on PyPI predates this work and does not carry the models below.

Extras, selected with uv sync --extra <name> or all at once with uv sync --all-extras:

Flash attention and the other fused kernels come through transformers[kernels], which the base install already carries, so there is no extra to select for them.

---

🚀 Usage

Models that transformers already ships load straight from their published repository:

import soundfile as sf
from transformers import AutoModelForTextToWaveform, AutoProcessor

model_id = "bosonai/higgs-tts-2-3b-base"

processor = AutoProcessor.from_pretrained(model_id)
model = AutoModelForTextToWaveform.from_pretrained(model_id).to("cuda")
processor.audio_tokenizer.to(model.device)

conversation = [
    {"role": "system", "content": [{"type": "text", "text": "Generate audio following instruction."}]},
    {"role": "user", "content": [{"type": "text", "text": "The sun rises in the east."}]},
]
inputs = processor.apply_chat_template(
    conversation,
    return_dict=True,
    tokenize=True,
    add_generation_prompt=True,
    sampling_rate=24000,
    return_tensors="pt",
).to(model.device)

audio_codes = model.generate(**inputs, max_new_tokens=1024)
waveform = processor.decode(audio_codes)
sf.write("output.wav", waveform.numpy(), processor.audio_tokenizer.config.sample_rate)

Models whose upstream release ships a bespoke weight layout are converted once through their folder's weight_conversion.convert, after which they load the same way. Each model's own README carries its conversion call, the generation arguments that are load bearing for it, its training objective, and what was not carried over from upstream.

---

📊 Models

Every model below loads real published weights and has been run against them. Follow the model name for its folder README, which documents its usage, its objective and its open items.

Voice Cloning

Reproduce the voice of a reference recording.

Voice Design

Build a voice from a natural language description, with no reference recording.

PromptTTS++ publishes no model repository. Its only public weights are bundled inside the Space linked above, which is what its weight_conversion.convert downloads.

Voice Editing

Change the voice of a recording, or rewrite part of it, while keeping the rest.

F5-TTS infills a masked span of an existing recording through its edit_mask argument. All three CosyVoice versions convert the voice of a recording while keeping its content, through source_speech_token_ids.

Vocoders and Codecs

Not text-to-speech models. These turn features or codes into a waveform, or a waveform into tokens, and the models above hold them as submodels.

Status legend

Year is the year the model was first published. An empty Paper cell means the release has no arXiv paper, only code and a model card. PROJECT.md carries the per-model verification evidence and the full list of open items, including one the Status column does not cover: Higgs TTS 3 reports 528 unexpected keys on load, all of them the codec copy bundled in its checkpoint.

---

🤝 Contributing

Issues and pull requests are welcome at github.com/LatentForge/VoiceStudio.

Two files in the repository root are the working documentation, and both are worth reading before opening a pull request. CLAUDE.md is the conventions document: how a model is migrated, what counts as verification, how files and comments are named, and the rules on dependencies and licence headers. PROJECT.md is the running status of the work, including every open item recorded against a model.

Areas where help is most useful:

• The items PROJECT.md records as open. The two needing a decision rather than code are CosyVoice v3's text normalizer, where upstream's ttsfrd and wetext are a closed wheel and a compiled grammar rather than inlinable logic, and its second released llm.rl.pt checkpoint, which trades character error rate for speaker similarity and has no way to be selected yet.
• Inference performance. Nothing here has been tuned for it, and the route is the transformers one, a static cache and a compiled graph selected through GenerationConfig, rather than a per-model capture. PROJECT.md has the detail.
• More models, migrated the way the existing nineteen were.

---

📝 License

Apache License 2.0. See LICENSE.

Each modeling_<model>.py also carries the licence header of the project its code came from, which is not always Apache 2.0.

The checkpoints are under their own licences, which are not this repository's, and several are more restrictive than the code that loads them. BreezeBlue/Breeze-TTS-2 ships a research and non-commercial licence, bosonai/higgs-tts-3-4b likewise, and FlashLabs/Chroma-4B is gated behind an access request. Review a checkpoint's licence before using it.

---

🙏 Acknowledgments

This repository is other people's research, brought under one API. The models come from:

• NVIDIA for BigVGAN
• BreezeBlue for Breeze TTS 2
• FlashLabs for Chroma
• FunAudioLLM for CosyVoice v1, v2 and v3
• Nari Labs for Dia and Dia2
• SWivid for F5-TTS and E2-TTS
• Boson AI for Higgs TTS 2 and Higgs TTS 3
• k2-fsa for OmniVoice
• Hugging Face for Parler-TTS
• LINE for PromptTTS++
• Qwen for Qwen3-TTS
• SparkAudio for Spark-TTS and BiCodec
• gemelo.ai for Vocos
• THU-HCSI for VoxInstruct

And the libraries the code is built out of:

• Hugging Face transformers, whose model classes almost every file here inherits from.
• PyTorch, with torchaudio and torchcodec behind the audio extra.
• librosa and NumPy.

---

🔗 Links

• Repository: github.com/LatentForge/VoiceStudio
• Group homepage: latentforge.github.io