Fish Audio TTS

A Rust implementation of text-to-speech using the S1-mini model from Fish Audio.

Overview

This crate provides a high-level API for generating speech from text using the OpenAudio S1-mini model, which is based on the Fish Speech architecture. The implementation includes:

• Tokenizer: Text tokenization using tiktoken-style BPE with semantic audio tokens
• Model: DualAR Transformer architecture for text-to-semantic token generation
• Audio Codec: DAC-based codec for converting semantic tokens to audio waveforms
• Automatic Model Download: Models are automatically downloaded from HuggingFace Hub

Features

• Zero-shot TTS: Generate speech from text without any voice samples
• Voice Cloning: Clone voices from reference audio samples (planned)
• Multi-language Support: Supports English, Chinese, Japanese, Korean, and more
• Multiple output formats: WAV, MP3 (via conversion)
• Automatic model downloading from HuggingFace Hub
• Support for HuggingFace token authentication via HF_TOKEN environment variable
• Support for using pre-downloaded model files

Installation

Add this to your Cargo.toml:

[dependencies]
fish-audio-tts = { git = "https://github.com/ayourtch/fish-audio-experiment" }

Feature Flags

• cuda: Enable CUDA GPU acceleration
• metal: Enable Metal GPU acceleration (macOS)
• accelerate: Enable Apple Accelerate framework

Quick Start

Automatic Model Download

The model is automatically downloaded from HuggingFace Hub on first use:

use fish_audio_tts::{download_model, TTSEngine, GenerationConfig, write_audio};
use candle_core::Device;

fn main() -> anyhow::Result<()> {
    // Download the model (cached after first download)
    let paths = download_model()?;
    
    // Create the TTS engine from downloaded model
    let engine = TTSEngine::from_model_paths(&paths, Device::Cpu)?;

    // Generate speech
    let config = GenerationConfig::default();
    let result = engine.synthesize("Hello, world!", &config)?;

    // Save the audio
    write_audio("output.wav", &result.audio)?;

    Ok(())
}

Or use the convenient from_hub method:

use fish_audio_tts::{TTSEngine, GenerationConfig, write_audio};
use candle_core::Device;

fn main() -> anyhow::Result<()> {
    // Create engine with automatic download
    let engine = TTSEngine::from_hub(Device::Cpu)?;

    // Generate speech
    let config = GenerationConfig::default();
    let result = engine.synthesize("Hello, world!", &config)?;

    // Save the audio
    write_audio("output.wav", &result.audio)?;

    Ok(())
}

Command Line Interface

# Download the model from HuggingFace
cargo run -- download

# Download with HuggingFace token for gated models
cargo run -- download --hf-token YOUR_TOKEN
# Or use the environment variable
HF_TOKEN=YOUR_TOKEN cargo run -- download

# Show information about available devices
cargo run -- info

# Synthesize speech from text (with automatic download)
cargo run -- synthesize --text "Hello, world!" --output output.wav --from-hub

# Synthesize using a pre-downloaded model
cargo run -- synthesize --text "Hello, world!" --output output.wav --model-path /path/to/model

# Synthesize from a text file
cargo run -- from-file --input story.txt --output narration.mp3 --from-hub

CLI Options

fish-tts synthesize [OPTIONS]

Options:
  -t, --text <TEXT>                     Text to synthesize
  -o, --output <OUTPUT>                 Output audio file path [default: output.wav]
      --temperature <TEMPERATURE>       Temperature for sampling (0.0-2.0) [default: 0.7]
      --top-p <TOP_P>                   Top-p (nucleus) sampling threshold [default: 0.8]
      --repetition-penalty <PENALTY>    Repetition penalty [default: 1.1]
      --max-tokens <MAX_TOKENS>         Maximum tokens to generate [default: 1024]
      --seed <SEED>                     Random seed for reproducibility
      --cpu                             Use CPU instead of GPU
      --from-hub                        Download and use model from HuggingFace Hub
      --model-path <MODEL_PATH>         Path to a local directory with pre-downloaded model files
      --hf-token <HF_TOKEN>             HuggingFace API token (can also use HF_TOKEN env var)

Environment Variables

• HF_TOKEN: HuggingFace API token for authentication. This is useful for accessing gated models or private repositories. The token can also be passed via the --hf-token CLI option.

Using Pre-downloaded Models

If you have already downloaded the model files, you can use them directly without downloading from HuggingFace:

use fish_audio_tts::{ModelPaths, TTSEngine, GenerationConfig, write_audio};
use candle_core::Device;

fn main() -> anyhow::Result<()> {
    // Use pre-downloaded model files
    let paths = ModelPaths::from_local("/path/to/model")?;
    
    // Create the TTS engine from local model
    let engine = TTSEngine::from_model_paths(&paths, Device::Cpu)?;

    // Generate speech
    let config = GenerationConfig::default();
    let result = engine.synthesize("Hello, world!", &config)?;

    // Save the audio
    write_audio("output.wav", &result.audio)?;

    Ok(())
}

The model directory should contain:

• config.json - Model configuration
• model.safetensors - Model weights
• tokenizer.tiktoken or tokenizer.json - Tokenizer file
• firefly_gan_vq.ckpt (optional) - Audio decoder

Using HuggingFace Token

For gated models or private repositories, you can provide a HuggingFace token:

use fish_audio_tts::{download_model_with_token, TTSEngine};
use candle_core::Device;

fn main() -> anyhow::Result<()> {
    // Use token from environment variable
    let token = std::env::var("HF_TOKEN").ok();
    
    // Download with token
    let paths = download_model_with_token(token)?;
    
    // Create the TTS engine
    let engine = TTSEngine::from_model_paths(&paths, Device::Cpu)?;

    Ok(())
}

Examples

Simple TTS Example

cargo run --example simple_tts -- "Hello, this is a test!"

Text File to MP3

cargo run --example text_file_to_mp3 -- input.txt output.mp3

Architecture

The Fish Audio TTS system consists of several components:

1. Tokenizer (src/tokenizer/): Converts text to tokens using tiktoken-style BPE tokenization with special tokens for audio semantics.

2. Text-to-Semantic Model (src/model/): DualAR Transformer that generates semantic audio tokens from text. Features:

   • RoPE (Rotary Position Embeddings)
   • RMSNorm (Root Mean Square Layer Normalization)
   • SwiGLU Feed-Forward Networks
   • Grouped Query Attention

3. Audio Codec (src/audio/): DAC-based codec that converts semantic tokens to audio waveforms.

4. TTS Engine (src/engine.rs): High-level API that orchestrates the complete TTS pipeline.

Model

This crate is designed to work with the OpenAudio S1-mini model:

• HuggingFace: https://huggingface.co/fishaudio/openaudio-s1-mini
• Sample Rate: 44100 Hz
• Number of Codebooks: 10

Original Repository

In the orig-repo/ directory, you'll find the original Fish Audio repository from https://github.com/fishaudio/fish-speech with the history removed to save space. This serves as the reference implementation for the Rust port.

Development

Building

# Build the library
cargo build

# Build with CUDA support
cargo build --features cuda

# Run tests
cargo test

Running Examples

# Simple TTS
cargo run --example simple_tts

# Text file to MP3
cargo run --example text_file_to_mp3 -- input.txt output.mp3

License

Apache-2.0

Acknowledgments

• Fish Audio for the original Fish Speech implementation
• Candle for the Rust ML framework