Vyvo-Multilingual-EN-FT-v0.1

An English voice-cloning TTS model — Vyvo/Vyvo-Multilingual-v0.1 fine-tuned for a single expressive English speaker. Give a short reference clip (+ its transcript) and a target text; the model speaks the text in that voice by pure language-model continuation.

• Backbone: Qwen3-0.6B + kyutai/mimi audio codec (32 codebooks, 24 kHz)
• Base model: Vyvo/Vyvo-Multilingual-v0.1
• Goal: sharply improve English intelligibility / WER for this voice over the multilingual base

Results

English Word Error Rate on 100 evaluation texts, transcribed with Qwen3-ASR-1.7B (lower = better):

Fine-tuning roughly halves the word error rate. Greedy decoding under-reports quality because of occasional early end-of-speech; with sampling + best-of-N the model reaches ~93 % word accuracy.

Recommended decoding

The model is a transcript-conditioned TTS: always pass the reference clip's transcript. Greedy can truncate long sentences, so for production use sampling (temperature ≈ 0.4–0.5, top_p = 0.9, repetition_penalty = 1.1) and, when quality matters, best-of-N (generate 4–6 candidates and keep the one whose ASR transcript best matches the target).

Usage

import soundfile as sf
import torch
import torchaudio
from transformers import (AutoModelForCausalLM, AutoTokenizer,
                          AutoFeatureExtractor, MimiModel)

REPO = "Vyvo/Vyvo-Multilingual-EN-FT-v0.1"
DEVICE = "cuda"

# Token layout (must match training)
BASE = 151669
NUM_CODEBOOKS, CODEBOOK_SIZE, AUDIO_OFFSET = 32, 2048, 10
SOS, EOS, SOH, EOH, SOA = 1, 2, 3, 4, 5   # special tokens, as offsets above BASE

tokenizer = AutoTokenizer.from_pretrained(REPO)
model = AutoModelForCausalLM.from_pretrained(REPO, dtype=torch.bfloat16).to(DEVICE).eval()
mimi = MimiModel.from_pretrained("kyutai/mimi").to(DEVICE).eval()
feature_extractor = AutoFeatureExtractor.from_pretrained("kyutai/mimi")


def encode_reference(path):
    wav, sr = sf.read(path, dtype="float32", always_2d=False)
    if wav.ndim > 1:
        wav = wav.mean(axis=1)
    target_sr = feature_extractor.sampling_rate
    if sr != target_sr:
        wav = torchaudio.functional.resample(
            torch.from_numpy(wav).unsqueeze(0), sr, target_sr).squeeze(0).numpy()
    inputs = feature_extractor(raw_audio=wav, sampling_rate=target_sr, return_tensors="pt")
    codes = mimi.encode(inputs["input_values"].to(DEVICE),
                        num_quantizers=NUM_CODEBOOKS).audio_codes[0].cpu()
    flat = codes.transpose(0, 1).reshape(-1).tolist()
    return [c + AUDIO_OFFSET + (i % NUM_CODEBOOKS) * CODEBOOK_SIZE + BASE
            for i, c in enumerate(flat)]


def build_prompt(ref_tokens, ref_text, target_text):
    text_ids = tokenizer(ref_text + " " + target_text, add_special_tokens=False).input_ids
    head = [BASE + SOH] + text_ids + [tokenizer.eos_token_id, BASE + EOH, BASE + SOA, BASE + SOS]
    return head + ref_tokens


def decode_audio(generated_ids):
    codes = []
    for i, tok in enumerate(generated_ids):
        v = tok - BASE - AUDIO_OFFSET - (i % NUM_CODEBOOKS) * CODEBOOK_SIZE
        if 0 <= v < CODEBOOK_SIZE:
            codes.append(v)
        else:
            break
    frames = len(codes) // NUM_CODEBOOKS
    codes = torch.tensor(codes[:frames * NUM_CODEBOOKS]).view(frames, NUM_CODEBOOKS)
    codes = codes.t().unsqueeze(0).to(DEVICE)
    return mimi.decode(codes).audio_values.squeeze().cpu().float().numpy()


reference_wav = "reference.wav"
reference_text = "text spoken in the reference clip"
target_text = "Hello, this is a test of the text to speech model."

prompt = build_prompt(encode_reference(reference_wav), reference_text, target_text)
input_ids = torch.tensor([prompt], device=DEVICE)
output = model.generate(
    input_ids, attention_mask=torch.ones_like(input_ids),
    max_new_tokens=9600, min_new_tokens=960,
    do_sample=True, temperature=0.45, top_p=0.9, repetition_penalty=1.1,
    eos_token_id=BASE + EOS, pad_token_id=tokenizer.eos_token_id)
audio = decode_audio(output[0, input_ids.shape[1]:].tolist())
sf.write("output.wav", audio, mimi.config.sampling_rate)

Training

Full fine-tune of every weight, continued from the base checkpoint (vocabulary already resized for the Mimi tokens).

A learning-rate sweep (1e-5 → 5e-5 → 1e-4) and a LoRA run were compared; full fine-tuning at 1e-4 won (LoRA reached WER 0.329, lr 5e-5 reached 0.310).

Limitations

• Requires the reference transcript. Transcript-free / zero-shot cloning collapses (the model speaks short, unrelated fragments).
• Single speaker / English only. Rare proper nouns are occasionally mis-rendered.
• Greedy decoding may truncate long sentences — see Recommended decoding.

License

Fine-tuned from Vyvo/Vyvo-Multilingual-v0.1. Released under the MIT License.