MLX-Audio

Text-to-Speech (TTS) and Speech-to-Text (STT) library based on Apple's MLX framework, optimized for Apple Silicon.

🚀 Two Versions Available

🌐 Web UI Version

• For Developers: Full API access, customizable, extensible
• Installation: Clone repo + run scripts
• Size: ~2GB (models downloaded separately)
• Best for: Development, integration, customization

📱 Mac App Version

• For End Users: One-click installation, no setup required
• Installation: Download 592MB All-in-One DMG
• Size: 592MB (models included)
• Best for: Quick usage, non-technical users

✨ Features

🌊 Real-time Streaming

• Stream-as-you-generate - First audio <500ms
• PCM format - Real-time Web Audio API playback
• Smart sentence splitting - Split by punctuation (,;:。！？)
• Performance monitoring - Real-time TTFB, playback time, data size

🚀 Performance Optimized

• Model preloading - Load 3 models to unified memory at startup
• Fine-grained splitting - Split by commas, periods for faster first byte
• Non-blocking playback - Stream while receiving, 250ms buffer

📊 Performance Metrics

Streaming Performance:

• First byte latency: <500ms
• Playback start: <500ms
• Real-time data increment display

🎯 Quick Start

Web UI Version

# Install dependencies
./scripts/setup.sh

# Start service
./scripts/start.sh

# Access
open http://localhost:8002

Mac App Version

1. Download the 592MB All-in-One DMG from Releases
2. Double-click to mount the DMG
3. Drag MLX-Audio.app to Applications folder
4. Launch from Applications or Launchpad
5. No additional setup required - models included!

🔧 Installation & Deployment

Web UI Version Requirements

• Python >=3.10
• MLX >=0.25.2
• FastAPI >=0.110.0
• Apple Silicon (M-series chip)

Detailed Setup

# Clone repository
git clone https://github.com/jiasunm/mlx-audio.git
cd mlx-audio

# Create virtual environment
python -m venv venv
source venv/bin/activate

# Install dependencies
pip install -r requirements.txt

# Download models (first run)
python -m mlx_audio.download_models

# Start server
uvicorn mlx_audio.main:app --host 0.0.0.0 --port 8002

⚙️ Configuration

# mlx_audio/config.py
preload_models = [
    "mlx-community/Kokoro-82M-bf16",      # TTS - Chinese/English/Japanese
    "mlx-community/VoxCPM1.5",            # TTS - Chinese/English bilingual
    "mlx-community/whisper-large-v3-turbo", # STT
]

📡 Usage Examples

TTS - Text to Speech

Streaming Mode (Recommended - stream while generating)

curl -X POST http://localhost:8002/v1/audio/speech/stream \
  -H "Content-Type: application/json" \
  -d '{
    "input": "Hello world, this is a streaming output test.",
    "lang_code": "a",
    "voice": "af_bella"
  }' -o output.pcm

Standard Mode (Generate complete WAV file)

curl -X POST http://localhost:8002/v1/audio/speech \
  -H "Content-Type: application/json" \
  -d '{
    "input": "Hello world",
    "lang_code": "a", 
    "voice": "af_bella",
    "speed": 1.0
  }' -o output.wav

STT - Speech to Text

curl -X POST http://localhost:8002/v1/audio/transcriptions \
  -F "file=@audio.wav" \
  -F "language=en" \
  -F "prompt=Technical terms, names, etc."

🎤 Supported Languages & Voices

Kokoro-82M (Recommended)

Chinese (lang_code: "z")

• zf_xiaobei (Xiaobei ♀)
• zf_xiaoni (Xiaoni ♀)
• zf_xiaoxiao (Xiaoxiao ♀)
• zm_yunjian (Yunjian ♂)

English (lang_code: "a"/"b")

• af_heart, af_nova, af_bella (American ♀)
• am_adam, am_michael (American ♂)
• bf_emma, bf_isabella (British ♀)
• bm_george, bm_lewis (British ♂)

Japanese (lang_code: "j")

• jf_alpha, jf_gongitsune (♀)
• jm_kumo (♂)

VoxCPM1.5 (Bilingual)

• Supports Chinese & English
• Auto language detection

🛠️ Tech Stack

• Core: Python 3.10+, Apple MLX Framework
• Web: FastAPI, Uvicorn, WebSockets
• Audio: librosa, soundfile, numpy
• Models: Kokoro-82M, VoxCPM1.5, Whisper-Turbo
• Frontend: HTML5, Web Audio API, JavaScript

🤝 Contributing

1. Fork the repository
2. Create feature branch (git checkout -b feature/amazing-feature)
3. Commit changes (git commit -m 'Add amazing feature')
4. Push to branch (git push origin feature/amazing-feature)
5. Open Pull Request

📝 Changelog

v1.0.0 (Latest) - 2026-01-02

• 🎉 Major Release: Production-ready version
• 📱 Mac App: All-in-One desktop application (592MB DMG)
• 🌍 12 Languages UI: Complete internationalization
• 🎨 Dynamic Spectrum: Real-time audio visualization
• 🎤 6 Whisper Models: Tiny to Large with 4-bit quantization
• 📝 Professional Prompt: 150+ technical terms for better accuracy
• 🔄 Async STT: Support for long audio files
• 💾 Audio Download: Download generated audio
• 🚀 Model Management: Load/unload models dynamically

v0.7.4 - Tauri Desktop App

• 📱 Tauri-based Mac application
• 🎯 All-in-One package with Python + Kokoro model
• ✅ No external dependencies
• 🌐 Complete i18n support

v0.5.0 - Web UI Enhancement

• 🌍 12 language UI support
• 🎨 Modern gradient theme
• 📊 Real-time performance metrics
• 🔧 Model preloading optimization

v0.4.0 - Advanced Features

• 🎨 Voice blending functionality
• 📊 Dynamic waveform visualization
• 🎤 Multiple Whisper model selection
• 📝 Subtitle generation (SRT/VTT)

v0.3.2

• ✨ Real-time streaming output with <500ms latency
• 🚀 Model preloading and performance optimization
• 📊 Performance monitoring and metrics
• 🌊 Smart sentence splitting for faster first byte

v0.3.0

• 🎯 Initial release with Kokoro-82M
• 🌐 Web UI and API endpoints
• 📱 Multi-language support

📄 License

This project is licensed under the MIT License - see the LICENSE file for details.

⭐ Star History

📱 Follow Us

---