Irodori-TTS-v4-Small

⚠️ IMPORTANT: NEW VERSION AVAILABLE

A minor update, Irodori-TTS-v4.1-Small, has been released! This new version features an improved Duration Predictor for better accuracy. We highly recommend using v4.1 instead of this model.

Irodori-TTS-v4-Small is a Japanese Text-to-Speech model based on a Rectified Flow Diffusion Transformer (RF-DiT) architecture. It combines Text (Input) + Reference Speech + Caption Text in a unified model, enabling zero-shot voice cloning, text-based voice design, and style-controlled voice cloning.

The model also supports emoji-based style and sound effect control. By inserting specific emojis into the input text, you can control speaking styles, emotions, and non-verbal vocal expressions in the generated audio.

🌟 Key Features

• Multi-modal Voice Design: Use reference audio for speaker identity and descriptive captions for voice characteristics, emotion, speaking style, and delivery.
• Long-reference Voice Cloning: Condition generation on one or more reference clips, with up to 120 seconds of combined reference audio.
• Flow Matching TTS: Rectified Flow Diffusion Transformer over continuous DACVAE latents for high-quality Japanese speech synthesis.
• Emoji-based Style Control: Embed emojis directly in the input text for granular control over the delivery and sound effects (e.g., laughter, coughing, sighs). See EMOJI_ANNOTATIONS.md for details.
• Integrated Watermarking: Integrates SilentCipher to apply robust, invisible audio watermarks directly to generated outputs, promoting responsible AI usage.

✨ What's New in v4

This version introduces several improvements over Irodori-TTS-600M-v3-VoiceDesign:

• Pretrained Japanese Text Encoder: Replaced the previous scratch-trained text and caption encoders with a shared, fine-tuned ModernBERT-ja backbone. This improves difficult Kanji reading and provides stronger pretrained language representations for both input text and Voice Design captions.
• Improved Text and Caption Understanding: The shared pretrained encoder strengthens linguistic understanding for text pronunciation and descriptive caption conditioning, while separate learned projectors adapt its representations to each conditioning path.
• Long Reference Support: Speaker patching and random-concatenation training extend reference conditioning to a combined maximum of 120 seconds. Multiple reference clips can be supplied and concatenated automatically.

---

🏗️ Architecture

The model (approximately 766M parameters) consists of five main components:

1. Shared Text/Caption Encoder: A fine-tuned Japanese ModernBERT backbone shared between the input-text and Voice Design caption paths.
2. Condition Projectors: Separate learned projectors map text and caption encoder representations into their respective TTS conditioning spaces.
3. Reference Latent Encoder: Encodes patched reference audio latents for speaker identity conditioning, supporting up to 120 seconds of combined reference audio.
4. Diffusion Transformer: Joint-attention DiT blocks combining text, reference, and caption conditioning with Low-Rank AdaLN, half-RoPE, and SwiGLU MLPs.
5. Duration Predictor: Predicts audio duration from encoded text and conditioning vectors using stacked SwiGLU MLP blocks.

Audio is represented as continuous latent sequences via the Aratako/Semantic-DACVAE-Japanese-32dim codec (32-dim), enabling high-quality 48kHz waveform reconstruction.

---

🎧 Audio Samples

Note: To clearly demonstrate the effect of captions, the samples within each group below were generated using the exact same random seed. The variations in delivery are purely the result of the changed prompts.

1. Pure Voice Design (Text + Caption)

Generate diverse voices and styles purely through descriptive text captions without any reference audio.

2. Style-Controlled Voice Cloning (Text + Caption + Ref Speech)

Clone a voice using reference audio, and dictate the specific emotion or delivery style using a caption.

3. Fully Controlled Generation (Text + Caption + Ref Speech + Emoji)

Combine all control vectors for maximum expressiveness, adding specific physiological sounds (sighs, coughs) or distinct nuances via emojis on top of the cloned and styled voice.

---

🚀 Usage

For inference code, installation instructions, and training scripts, please refer to the GitHub repository:

👉 GitHub: Aratako/Irodori-TTS

For lower-memory inference, torchao INT8, INT4, and FP8 variants are available at Aratako/Irodori-TTS-v4-Small-Quantized.

Long-reference Voice Cloning

For long-reference voice cloning, using multiple shorter clips from the same speaker is recommended. v4-Small was trained by randomly concatenating multiple short utterances from each speaker, and the reference-length benchmark below follows the same construction. Inference also accepts a single uninterrupted long recording, but the benefit of that input format has not been evaluated and may differ from the reported results.

📊 Benchmarks

The Irodori-TTS-v4-Small and Irodori-TTS-600M-v3-VoiceDesign results below were measured under matched inference settings. The five-seed evaluations use the consecutive base sampling seeds 0 through 4. Unless otherwise noted, values are the mean and population standard deviation across these five seeds.

The Joyo Kanji Yomi Benchmark, JSUT, Coco-Nut, and JVS datasets were not included in the training data.

Japanese Reading

Japanese reading was evaluated on the Joyo Kanji Yomi Benchmark and JSUT BASIC5000. Irodori generation used no reference audio or Voice Design caption, FP32 inference, 40 RF steps, and text CFG 3.0. The published T5Gemma-TTS and Sarashina2.2-TTS Stage 2 values are taken from the Sarashina2.2-TTS paper.

Joyo Kanji Yomi Benchmark

v4-Small improved the target-kanji Kana-CER metrics in every seed. Its sentence-level Kana-CER and Standard CER were slightly worse than v3, so the improvement does not apply uniformly to every reading metric.

JSUT BASIC5000

Note: These published values are contextual references, not a controlled model ranking. T5Gemma-TTS and Sarashina2.2-TTS use an unpublished fixed reading-style reference speech prompt and its transcript, whereas the Irodori runs use neither reference audio nor a caption. Model architectures and inference settings also differ.

Voice Design

Voice Design prompt adherence was evaluated using all 2,890 public voice descriptions from the Coco-Nut test set. Each description was paired with the same short, medium, and long reading texts, producing 8,670 clips per model without reference audio and using one deterministic seed per description/text pair. Gemini 3.6 Flash independently scored each clip from 1 to 5 for agreement between the requested voice description and the generated speech.

Unlike the other benchmarks, this evaluation uses a single generation run with base seed 0. The table therefore reports aggregates over the evaluated clips rather than a mean and standard deviation across five synthesis seeds.

This is a lightweight, cost-conscious internal comparison using a single automatic judge without human validation. It is not intended as a research-grade benchmark or as standalone evidence for academic claims.

See VOICE_DESIGN_BENCHMARK.md for the complete protocol, scoring rubric, results, and limitations.

Voice Cloning and Reference Length

Voice cloning was evaluated on JVS using all 100 speakers, five target texts, and five synthesis seeds. For v4-Small, independently encoded reference utterances from each speaker were concatenated into nested one-clip, approximately 30-second, approximately 60-second, and 120-second conditions. Similarity was measured against a fixed centroid of ten held-out natural utterances per speaker, so longer conditioning references did not change the scoring target.

v4-Small gains substantial speaker similarity from longer references, with most of the improvement already present at approximately 30 seconds. With only one short clip, v3 achieved higher similarity. This reduction in one-clip cloning performance is considered a tradeoff of changing speaker-conditioning training from primarily single clips to random concatenation for long-reference support. ERes2Net, used as a second independently trained speaker encoder, showed the same pattern. Extending the v4 reference from one clip to 120 seconds improved CAM++ cosine by +0.1143, with a speaker-bootstrap 95% interval of [+0.1058, +0.1230].

See VOICE_CLONING_BENCHMARK.md for the dataset, reference construction, generation conditions, complete CAM++/ERes2Net/CER results, and limitations.

📊 Training Data & Annotation

The model was trained on an expanded, high-quality Japanese speech dataset. To enable the multi-modal Voice Design functionality, the training data was enriched with comprehensive text captions describing the audio characteristics.

The emoji annotations and initial text captions were generated and labeled using a fine-tuned model based on Qwen/Qwen3-Omni-30B-A3B-Instruct. Subsequently, the text captions were rephrased and refined using Qwen/Qwen3.5-35B-A3B.

⚠️ Limitations

• Japanese Only: This model currently supports Japanese text input only.
• Short-reference Voice Cloning: With only one short reference clip, objective speaker similarity was modestly lower than v3 in the evaluation. Approximately 30 seconds or more of reasonably clean reference speech is recommended when available.
• Long-reference Composition: Training and long-reference evaluation used multiple short utterances from the same speaker concatenated together. A single uninterrupted long recording is supported as input, but its effect has not been evaluated.
• Conditioning Conflicts: When using both reference audio and a text caption, contradictory instructions may result in unstable audio quality, unnatural artifacts, or one condition overriding the other. For optimal results, use the caption to guide emotion, style, or environment while keeping the base voice characteristics aligned with the reference audio.
• Prompt Adherence: While the model generally follows caption instructions, highly complex or contradictory descriptions may produce inconsistent results.
• Emoji Control: The effect of emoji-based control may vary depending on context and is not always perfectly consistent.
• Kanji Reading: Difficult Kanji reading has improved over v3, but uncommon names, specialized terminology, and context-dependent readings may still be pronounced incorrectly.
• Evaluation Scope: No large-scale human MOS, naturalness, prompt-adherence, or speaker-similarity evaluation was conducted. Automatic benchmark scores do not fully represent human perception.

📜 License & Ethical Restrictions

License

This model is released under MIT.

Ethical Restrictions

In addition to the license terms, the following ethical restrictions apply:

1. No Impersonation: Do not use this model to clone or impersonate the voice of any individual (e.g., voice actors, celebrities, public figures) without their explicit consent.
2. No Misinformation: Do not use this model to generate deepfakes or synthetic speech intended to mislead others or spread misinformation.
3. Voice Generation Disclaimer: When generating speech purely from text or captions without using reference audio, it is possible that the generated voice may coincidentally resemble that of a real person. This is strictly a probabilistic artifact within the latent space. The model was not trained with the intent of reproducing specific individuals.
4. Liability Disclaimer: The developers assume no liability for any misuse of this model. Users are solely responsible for ensuring their use of the generated content complies with applicable laws and regulations in their jurisdiction.

🙏 Acknowledgments

This project builds upon the following works:

• Echo-TTS — Architecture and training design reference
• DACVAE — Audio VAE
• sbintuitions/modernbert-ja-310m — Pretrained Japanese text and caption encoder
• SilentCipher — Audio watermarking integration

We would also like to extend our special thanks to Respair for the inspiration behind the emoji annotation feature, and to gabrielclark3330 and kikouousya for supporting this project.

🖊️ Citation

If you use Irodori-TTS in your research or project, please cite it as follows:

@misc{irodori-tts-v4-small,
  author = {Chihiro Arata},
  title = {Irodori-TTS: A Flow Matching-based Text-to-Speech Model with Emoji-driven Style Control},
  year = {2026},
  publisher = {Hugging Face},
  journal = {Hugging Face repository},
  howpublished = {\url{https://huggingface.co/Aratako/Irodori-TTS-v4-Small}}
}