Models

• kani-tts-400m-en - English

• kani-tts-400m-zh - Chinese

• kani-tts-400m-de - German

• kani-tts-400m-ar - Arabic

• kani-tts-400m-es - Spanish

• kani-tts-400m-ko - Korean

• kani-tts-370m-expo2025-osaka-ja - Japanese

• kani-tts-400m-0.3-pt - Pretrained checkpoint v0.3

• kani-tts-370m multilingual - English, Spanish, Chinese, German, Korean, Arabic

• kani-tts-370m-mlx - Multilingual model for Apple Silicon

• kani-tts-450m-0.2-pt - Pretrained checkpoint v0.2 for posttraining and fine-tuning on custom datasets.

• nemo-nano-codec-22khz-0.6kbps-12.5fps-MLX - MLX implementation of NVIDIA NeMo NanoCodec, a lightweight neural audio codec.

Notes:

• Primarily optimized for English
• Performance degrades with inputs >1000 tokens
• Limited emotional expressivity without fine-tuning

Inference

Kani TTS offers multiple inference options optimized for different hardware:

Basic Example (GPU/CPU)

The basic inference example runs on both GPU and CPU, making it accessible for various hardware setups. Check the examples/basic in this repository for getting started. You can use the PyPi package kani-tts. More detailes...

vLLM (NVIDIA GPU)

For high-performance inference on NVIDIA GPUs, use KaniTTS-vLLM. This option is super fast and provides an OpenAI compatible API, making it easy to integrate with existing tools and workflows.

MLX (Apple Silicon)

For Apple Silicon users, we provide an optimized KaniTTS-MLX that takes full advantage of the unified memory architecture and Neural Engine on M1/M2/M3 chips.

NeMo NanoCodec

• Base Model: NVIDIA NeMo NanoCodec
• License: NVIDIA Open Model License
• Sample Rate: 22.05 kHz
• Purpose: Neural audio compression/decompression for TTS pipelines

---

GPU Benchmark Results

Lower RTF is better (< 1.0 means faster than real-time). Benchmarks conducted on Vast AI.

---

Dataset Preparation

1. Audio Dataset Collection

You can prepare your audio dataset using Datamio, our active community members. Datamio provides tools to help you collect, organize, and manage high-quality audio datasets for TTS training.

2. Audio Processing Pipeline

After collecting your raw audio dataset, you need to process it for training. Check out this audio processing pipeline: nano-codec-dataset-pipeline

This pipeline handles:

• Audio preprocessing and normalization
• Feature extraction
• Dataset formatting for training
• Quality validation

---

Finetuning

For finetuning KaniTTS on your own dataset, check out this comprehensive finetuning pipeline: KaniTTS-Finetune-pipeline

This pipeline provides:

• Step-by-step finetuning guides
• Configuration templates
• Training scripts optimized for different hardware setups
• Evaluation code to assess model performance
• Best practices for achieving high-quality results

---

App Examples

• ComfyUI node by WildAi

• NextJS basic app. It uses the OpenAI npm package to connect to the OpenAI-compatible server API provided by kanitts-vllm.

• Livekit Agent - A real-time voice AI assistant built with LiveKit Agents framework, featuring speech-to-text, language processing, and text-to-speech capabilities.

---

Areas of improvement

We're continuously working to enhance KaniTTS. Here are key areas where we're focusing our efforts:

Core Architecture

• Create new LLM for TTS exclusively - Develop a specialized LLM designed specifically for text-to-speech generation, optimized for audio token prediction rather than adapted from general-purpose LLMs

Model Enhancements

• Add more languages - Expand support beyond the current languages to cover more language families and dialects
• Add more speakers - Increase speaker diversity with different accents, age groups, and voice characteristics
• Voice cloning examples - Provide tutorials and code examples for cloning custom voices from audio samples

Audio Codec Improvements

• Fine-tune codec - Optimize the existing NanoCodec for better audio quality and compression efficiency
• Create new codec - Develop a next-generation neural audio codec with improved naturalness and lower latency

Dataset Development

Build and release high-quality, diverse audio datasets for training and fine-tuning.

• Multi-speaker datasets across different languages
• Domain-specific datasets (conversational, storytelling, professional voice-over)
• Benchmark datasets for evaluation

If you're interested in contributing to any of these areas, please check our Contributing section and join our Discord server.

---

License

Apache 2. See LICENSE file for details.

---

Contributing

We're open for community contributions! KaniTTS is built with the community, and we welcome contributions of all kinds:

• Code contributions - Bug fixes, new features, optimizations, and documentation improvements
• Model contributions - Fine-tuned models, voice clones, and language-specific adaptations
• Dataset contributions - High-quality audio datasets for training and evaluation
• Examples and tutorials - Integration examples, use cases, and guides
• Bug reports and feature requests - Help us improve by reporting issues and suggesting enhancements

How to contribute:

1. Check our Areas of improvement section for current priorities
2. Join our Discord server to discuss ideas and get support
3. Submit issues or pull requests on GitHub
4. Share your projects and use cases with the community