Irodori Character Voice

EN | JA

Irodori Character Voice is a model based on Irodori-TTS that generates speech conditioned on character images. Unlike the standard Irodori-TTS models, which use reference audio or Voice Design captions as conditioning, this method takes a character image as input and uses image-derived features to control the direction of the voice quality and speaking style.

Links

Original project: Aratako/Irodori-TTS

Demo

Install dependencies and launch the Gradio UI for character-image-conditioned inference.

uv sync
uv run python gradio_app_character.py --server-name 0.0.0.0 --server-port 7862

Open http://localhost:7862 in your browser.

Generated wav files are saved under gradio_outputs_character/.

You can also run inference from the CLI.

uv run python infer.py \
  --hf-checkpoint p1atdev/Irodori-TTS-500M-v2-Character-Voice-Tagger \
  --text "今日はいい天気ですね。" \
  --character-image samples/male-1.webp \
  --no-ref \
  --output-wav outputs/male-1.wav

Samples

See Project page.

Citation

@misc{character-voice-control,
  author = {Tingrui Zhou and Keiji Yanai},
  title = {A Character's Look Speaks Volumes: Character Image-Conditioned Speaker Style Control for Japanese Text-to-Speech},
  year = {2026},
  eprint={TODO},
  archivePrefix={arXiv},
  primaryClass={cs.SD},
  url={TODO}
}

@misc{irodori-tts,
  author = {Chihiro Arata},
  title = {Irodori-TTS: A Flow Matching-based Text-to-Speech Model with Emoji-driven Style Control},
  year = {2026},
  publisher = {GitHub},
  journal = {GitHub repository},
  howpublished = {\url{https://github.com/Aratako/Irodori-TTS}}
}

---

Irodori-TTS

Training and inference code for Irodori-TTS, a Flow Matching-based Text-to-Speech model. The architecture and training design largely follow Echo-TTS, using DACVAE continuous latents as the generation target.

[!IMPORTANT] main tracks the v2 codebase and is intended for use with the Irodori-TTS-500M-v2 and Irodori-TTS-500M-v2-VoiceDesign model releases. If you need the previous v1 code, use the v1 tag. v1 and v2 checkpoints / preprocessing are not compatible across versions. The previous public v1 model is available at Aratako/Irodori-TTS-500M.

For model weights and audio samples, please refer to the base model card and the VoiceDesign model card.

Features

• Flow Matching TTS: Rectified Flow Diffusion Transformer (RF-DiT) over continuous DACVAE latents
• Voice Cloning: Zero-shot voice cloning from reference audio
• Voice Design: Caption-conditioned style control
• Multi-GPU Training: Distributed training via uv run torchrun with gradient accumulation, mixed precision (bf16), and W&B logging
• PEFT LoRA Fine-Tuning: Parameter-efficient adaptation with PEFT/LoRA for released checkpoints
• Flexible Inference: CLI, Gradio Web UI, and HuggingFace Hub checkpoint support

Architecture

The v2 codebase supports two closely related checkpoint families:

1. Base model (Aratako/Irodori-TTS-500M-v2): Text encoder + reference latent encoder + diffusion transformer. The reference latent encoder consumes patched DACVAE latents from reference audio for speaker/style conditioning.
2. VoiceDesign model (Aratako/Irodori-TTS-500M-v2-VoiceDesign): Text encoder + caption encoder + diffusion transformer. The caption encoder consumes style-control text and the speaker/reference branch is disabled.

Shared building blocks:

1. Text Encoder: Token embeddings initialized from a pretrained LLM, followed by self-attention + SwiGLU transformer layers with RoPE
2. Condition Encoder: Either a reference latent encoder for the base model or a caption encoder for the VoiceDesign model
3. Diffusion Transformer: Joint-attention DiT blocks with Low-Rank AdaLN (timestep-conditioned adaptive layer normalization), half-RoPE, and SwiGLU MLPs

Audio is represented as continuous latent sequences via the codec configured by the checkpoint. v2 uses the 32-dim Semantic-DACVAE-Japanese-32dim codec for 48kHz waveform reconstruction.

Installation

git clone https://github.com/Aratako/Irodori-TTS.git
cd Irodori-TTS
uv sync

Note: For Linux/Windows with CUDA, PyTorch is automatically installed from the cu128 index. For macOS (MPS) or CPU-only usage, uv sync will install the default PyTorch build.

Quick Start

Simple Inference

uv run python infer.py \
  --hf-checkpoint Aratako/Irodori-TTS-500M-v2 \
  --text "今日はいい天気ですね。" \
  --ref-wav path/to/reference.wav \
  --output-wav outputs/sample.wav

Inference without Reference Audio

uv run python infer.py \
  --hf-checkpoint Aratako/Irodori-TTS-500M-v2 \
  --text "今日はいい天気ですね。" \
  --no-ref \
  --output-wav outputs/sample.wav

VoiceDesign Inference

uv run python infer.py \
  --hf-checkpoint Aratako/Irodori-TTS-500M-v2-VoiceDesign \
  --text "今日はいい天気ですね。" \
  --caption "落ち着いた女性の声で、近い距離感でやわらかく自然に読み上げてください。" \
  --no-ref \
  --output-wav outputs/sample_voice_design.wav

Gradio Web UI

uv run python gradio_app.py --server-name 0.0.0.0 --server-port 7860

Then access the UI at http://localhost:7860. The hosted v2 demo is available at Aratako/Irodori-TTS-500M-v2-Demo.

For the VoiceDesign checkpoint, use the dedicated UI:

uv run python gradio_app_voicedesign.py --server-name 0.0.0.0 --server-port 7861

The hosted VoiceDesign demo is available at Aratako/Irodori-TTS-500M-v2-VoiceDesign-Demo.

gradio_app.py is for Aratako/Irodori-TTS-500M-v2. gradio_app_voicedesign.py is for Aratako/Irodori-TTS-500M-v2-VoiceDesign.

Inference

CLI

uv run python infer.py \
  --hf-checkpoint Aratako/Irodori-TTS-500M-v2 \
  --text "今日はいい天気ですね。" \
  --ref-wav path/to/reference.wav \
  --output-wav outputs/sample.wav

Local checkpoints (.pt or .safetensors) are also supported:

uv run python infer.py \
  --checkpoint outputs/checkpoint_final.safetensors \
  --text "今日はいい天気ですね。" \
  --ref-wav path/to/reference.wav \
  --output-wav outputs/sample.wav

VoiceDesign checkpoints also support caption conditioning:

uv run python infer.py \
  --hf-checkpoint Aratako/Irodori-TTS-500M-v2-VoiceDesign \
  --text "今日はいい天気ですね。" \
  --caption "落ち着いた、近い距離感の女性話者" \
  --no-ref \
  --output-wav outputs/sample_voice_design.wav

Inference Parameters

Training

1. Prepare Manifest (Precompute DACVAE Latents)

Encodes audio from a Hugging Face dataset into DACVAE latents and produces a JSONL manifest for training.

uv run python prepare_manifest.py \
  --dataset myorg/my_dataset \
  --split train \
  --audio-column audio \
  --text-column text \
  --output-manifest data/train_manifest.jsonl \
  --latent-dir data/latents \
  --device cuda

To include speaker_id in the manifest (for speaker-conditioned training):

uv run python prepare_manifest.py \
  --dataset myorg/my_dataset \
  --split train \
  --audio-column audio \
  --text-column text \
  --speaker-column speaker \
  --output-manifest data/train_manifest.jsonl \
  --latent-dir data/latents \
  --device cuda

To include caption in the manifest (for caption-conditioned voice design training):

uv run python prepare_manifest.py \
  --dataset myorg/my_dataset \
  --split train \
  --audio-column audio \
  --text-column text \
  --caption-column caption \
  --speaker-column speaker \
  --output-manifest data/train_manifest.jsonl \
  --latent-dir data/latents \
  --device cuda

When training the caption-conditioned voice-design model, speaker_id is optional. The voice-design path disables speaker/reference conditioning and learns from text + caption.

This produces a JSONL manifest with entries like:

{"text": "こんにちは", "caption": "落ち着いた、近い距離感の女性話者", "latent_path": "data/latents/00001.pt", "speaker_id": "myorg/my_dataset:speaker_001", "num_frames": 750}

2. Training

Single-GPU training:

uv run python train.py \
  --config configs/train_500m_v2.yaml \
  --manifest data/train_manifest.jsonl \
  --output-dir outputs/irodori_tts

VoiceDesign training uses a dedicated config:

uv run python train.py \
  --config configs/train_500m_v2_voice_design.yaml \
  --manifest data/train_manifest.jsonl \
  --output-dir outputs/irodori_tts_voice_design

configs/train_500m_v2_voice_design.yaml sets use_caption_condition: true and disables the speaker/reference branch. Caption-free configs continue to use speaker conditioning when speaker_id / reference inputs are available.

The VoiceDesign config also enables caption_warmup: true for optional caption-branch warmup. warmup_steps controls the LR scheduler, while caption_warmup_steps controls how long non-caption gradients are discarded before normal joint training resumes.

Multi-GPU DDP training:

uv run torchrun --nproc_per_node 4 train.py \
  --config configs/train_500m_v2.yaml \
  --manifest data/train_manifest.jsonl \
  --output-dir outputs/irodori_tts \
  --device cuda

Training supports YAML config files with model and train sections. CLI arguments take precedence over YAML values. See uv run python train.py --help for all available options.

Fine-Tuning from Released Weights

Start a new training run from released inference weights (.safetensors). This initializes only the model weights; optimizer / scheduler state starts fresh.

uv run python train.py \
  --config configs/train_500m_v2.yaml \
  --manifest data/train_manifest.jsonl \
  --output-dir outputs/irodori_tts_ft \
  --init-checkpoint path/to/Irodori-TTS-500M-v2.safetensors

LoRA fine-tuning:

uv run python train.py \
  --config configs/train_500m_v2_lora.yaml \
  --manifest data/train_manifest.jsonl \
  --output-dir outputs/irodori_tts_lora \
  --init-checkpoint path/to/Irodori-TTS-500M-v2.safetensors

Caption-conditioned voice-design LoRA fine-tuning:

uv run python train.py \
  --config configs/train_500m_v2_voice_design_lora.yaml \
  --manifest data/train_manifest.jsonl \
  --output-dir outputs/irodori_tts_voice_design_lora \
  --init-checkpoint path/to/Irodori-TTS-500M-v2.safetensors

Available LoRA target presets:

• text_attn_mlp: text encoder attention + attention gate + MLP
• caption_attn_mlp: caption encoder attention + attention gate + MLP
• speaker_attn_mlp: speaker encoder attention + attention gate + MLP, plus speaker_encoder.in_proj
• diffusion_attn: diffusion attention only, including text/speaker/caption context KV and attention gate
• diffusion_attn_mlp: diffusion_attn + diffusion MLP
• all_attn: all attention blocks across text/caption/speaker/diffusion, including attention gates
• diffusion_full: diffusion stack broadly: cond_module, in_proj/out_proj, diffusion attention, diffusion MLP, and AdaLN
• adaln: diffusion-block AdaLN layers only
• conditioning: conditioning-side projections only: cond_module, speaker_encoder.in_proj, and diffusion context KV projections
• all_attn_mlp: all_attn + text/caption/speaker/diffusion MLP, plus speaker_encoder.in_proj
• all_linear: all nn.Linear layers in the model; embeddings and norm weights are not included

--lora-target-modules also accepts a regex string or a comma-separated list of module suffixes. Resume automatically restores the saved LoRA config from the training checkpoint unless you explicitly override it.

When --lora is enabled, checkpoints are saved as adapter-only directories containing PEFT adapter weights plus trainer state for resume.

Resuming Interrupted Training

Resume an existing training run from a training checkpoint. Full-model runs use .pt; LoRA runs use checkpoint directories. Both restore optimizer, scheduler, and step state.

uv run python train.py \
  --config configs/train_500m_v2.yaml \
  --manifest data/train_manifest.jsonl \
  --output-dir outputs/irodori_tts \
  --resume outputs/irodori_tts/checkpoint_0010000.pt

LoRA resume example:

uv run python train.py \
  --config configs/train_500m_v2_lora.yaml \
  --manifest data/train_manifest.jsonl \
  --output-dir outputs/irodori_tts_lora \
  --resume outputs/irodori_tts_lora/checkpoint_0010000

If you move a LoRA checkpoint to another environment and the original base-checkpoint path is no longer valid, pass --init-checkpoint path/to/base_model.safetensors together with --resume to override the saved base-model path.

3. Checkpoint Conversion

Convert a training checkpoint to inference-only safetensors format:

uv run python convert_checkpoint_to_safetensors.py outputs/checkpoint_final.pt

LoRA adapter checkpoints can also be converted directly:

uv run python convert_checkpoint_to_safetensors.py outputs/irodori_tts_lora/checkpoint_final

LoRA adapter checkpoints are merged into the base model automatically during conversion, so the exported .safetensors file is directly usable for inference.

Project Structure

Irodori-TTS/
├── train.py                    # Training entry point (DDP support)
├── infer.py                    # CLI inference
├── gradio_app.py               # Gradio web UI
├── gradio_app_voicedesign.py   # Gradio web UI for VoiceDesign checkpoints
├── prepare_manifest.py         # Dataset -> DACVAE latent preprocessing
├── convert_checkpoint_to_safetensors.py  # Checkpoint converter
│
├── irodori_tts/                # Core library
│   ├── model.py                # TextToLatentRFDiT architecture
│   ├── rf.py                   # Rectified Flow utilities & Euler CFG sampling
│   ├── codec.py                # DACVAE codec wrapper
│   ├── dataset.py              # Dataset and collator
│   ├── tokenizer.py            # Pretrained LLM tokenizer wrapper
│   ├── config.py               # Model / Train / Sampling config dataclasses
│   ├── inference_runtime.py    # Cached, thread-safe inference runtime
│   ├── lora.py                 # PEFT LoRA integration helpers
│   ├── text_normalization.py   # Japanese text normalization
│   ├── optim.py                # Muon + AdamW optimizer
│   └── progress.py             # Training progress tracker
│
└── configs/
    ├── train_500m_v2.yaml                    # 500M v2 model config
    ├── train_500m_v2_lora.yaml               # 500M v2 LoRA fine-tuning config
    ├── train_500m_v2_voice_design.yaml       # 500M v2 VoiceDesign full fine-tuning config
    ├── train_500m_v2_voice_design_lora.yaml  # 500M v2 VoiceDesign LoRA fine-tuning config
    ├── train_500m.yaml                       # 500M v1 model config
    └── train_2.5b.yaml                       # 2.5B parameter model config

License

• Code: MIT License
• Model Weights: Please refer to the base model card and the VoiceDesign model card for licensing details

Acknowledgments

This project builds upon the following works:

• Echo-TTS — Architecture and training design reference
• DACVAE — Audio VAE

Citation

@misc{irodori-tts,
  author = {Chihiro Arata},
  title = {Irodori-TTS: A Flow Matching-based Text-to-Speech Model with Emoji-driven Style Control},
  year = {2026},
  publisher = {GitHub},
  journal = {GitHub repository},
  howpublished = {\url{https://github.com/Aratako/Irodori-TTS}}
}