SharpAudio

SharpAudio is a high-performance Text-to-Speech and Speech-to-Text REST API with OpenAI-compatible endpoints, highly vibe-coded:blush:. Built with .NET 10, it features a Vue.js frontend, Docker containerization, and support for multiple state-of-the-art TTS models (Kokoro, Supertonic-3) and ASR models (Whisper, Nemotron). Which task types a given deployment serves — TTS, ASR, or both — is controlled by a single SERVER_MODE environment variable.

✨ Features

• 🎯 OpenAI-Compatible API - Drop-in replacement for OpenAI's /v1/audio/speech and /v1/audio/transcriptions endpoints
• 🚀 High Performance - Custom ONNX inference engine with direct OnnxRuntime integration
• 🎙️ Speech-to-Text (ASR) - Whisper (whisper.cpp/GGML) and NVIDIA Nemotron 3.5 (cache-aware streaming FastConformer-RNNT, ONNX) transcription engines
• 💾 Worker Process Architecture - OS-level memory isolation with automatic cleanup and guaranteed resource reclamation, for both TTS and ASR workers
• 🌍 Multilingual - Support for 3+ TTS languages and 35+ ASR languages including English, Japanese, Chinese, Spanish, French, and more
• 🎨 Multiple TTS Models - Kokoro (quantized & full precision) and Supertonic-3 models
• 🎭 Rich Voice Library - 5+ speaker voices with diverse characteristics
• 🔀 Configurable Server Mode - Run a single instance as TTS-only, ASR-only, or both via SERVER_MODE
• 🔧 Production Ready - Built-in health checks, automatic model management, and graceful degradation
• 📊 Interactive Docs - Swagger/OpenAPI documentation included
• 🎨 Modern UI - Vue.js frontend that adapts to the running server's capabilities (TTS tab, ASR tab, or both)
• 🐳 Docker Native - Multi-stage builds with optimized containerization, selectable per-mode via docker build --target
• 🧵 Thread-Safe - Advanced concurrency handling for multi-tenant deployments

🚀 Quick Start

Using Docker Compose (Recommended)

# Clone the repository
git clone https://github.com/Fhrozen/SharpAudio.git
cd SharpAudio

# Start the service
docker compose up -d

# Access the web interface
open http://localhost:5768

# Or use the API directly
curl -X POST http://localhost:5768/v1/audio/speech \
  -H 'Content-Type: application/json' \
  -d '{
    "model": "kokoro-q4",
    "input": "Hello world, this is a test of SharpAudio.",
    "voice": "af_bella",
    "speed": 1.0
  }' \
  --output speech.wav

The service will automatically download required models from Hugging Face on first startup.

By default the container only serves TTS. To enable Speech-to-Text as well, set SERVER_MODE:

# tts (default) | asr | both
SERVER_MODE=both docker compose up -d

# Transcribe an audio file (Whisper by default)
curl -X POST http://localhost:5768/v1/audio/transcriptions \
  -F file=@sample.wav \
  -F model=whisper-base

📚 Documentation

• Architecture - System design and components
• API Reference - Detailed endpoint documentation
• Configuration - Environment variables and settings
• Development Guide - Local development setup
• Deployment Guide - Production deployment strategies
• Models - TTS models and voice documentation
• Troubleshooting - Common issues and solutions
• Text Sanitization - Thread safety and text processing

🎯 Core Technologies

• Backend: .NET 10 with ASP.NET Core Minimal APIs
• Frontend: Vue.js 3.5 with TypeScript
• TTS Engine: Microsoft.ML.OnnxRuntime + espeak-ng
• ASR Engines: Whisper.net (whisper.cpp/GGML) & Microsoft.ML.OnnxRuntime (Nemotron)
• Models: Kokoro-82M & Supertonic-3 (TTS, ONNX); Whisper & Nemotron 3.5 (ASR, GGML/ONNX)
• Containerization: Docker with multi-stage builds
• Build Tool: pnpm for frontend dependencies

🔌 API Endpoints

🎭 Supported Models

Text-to-Speech

Speech-to-Text

See Models Documentation for detailed information about each model.

🌍 Supported Languages

English (US/GB), Spanish, French, Hindi, Italian, Japanese, Portuguese (BR), Chinese (CN), Korean, German, Dutch, Arabic, Russian, Turkish, Polish, Swedish, Danish, Norwegian, Finnish, Greek, Czech, Romanian, Hungarian, Thai, Vietnamese, Indonesian, Hebrew, Ukrainian, and more.

⚙️ Configuration

Key environment variables:

HTTP_PORT=5768                        # HTTP port
SERVER_MODE=tts                       # tts (default) | asr | both
MODEL_CACHE_DIR=/cache                # Model cache directory
ESPEAK_DATA_DIR=/app/assets/espeak-ng-data  # espeak-ng data
MODEL_IDLE_TIMEOUT_SECONDS=60         # Model unload timeout

See Configuration Guide for all options.

🛠️ Development

Local Development (without Docker)

# Backend
cd src/SharpAudio.Api
dotnet restore
dotnet run

# Frontend
cd frontend
corepack enable pnpm
pnpm install
pnpm dev

See Development Guide for detailed setup instructions.

🧪 Testing

# Run all tests
./tests/run-tests.sh

# Run specific test suite
dotnet test tests/SharpAudio.Api.Tests
dotnet test tests/SharpAudio.Api.IntegrationTests

# Opt-in: real-model circular TTS->ASR tests (downloads GB-scale weights, skipped by default and
# excluded from CI)
ASR_MODEL_TESTS=1 dotnet test tests/SharpAudio.Api.IntegrationTests --filter "Category=AsrModelTests"
# or
./tests/run-tests.sh asr-model-tests

📦 Production Deployment

SharpAudio is production-ready with:

• Automatic model memory management
• Graceful degradation on failures
• Health check endpoints
• Request validation and error handling
• Thread-safe concurrent request processing

See Deployment Guide for production deployment strategies.

🤝 Contributing

Contributions are welcome! Please feel free to submit a Pull Request.

📄 License

See LICENSE file for details.

🙏 Acknowledgments

• Kokoro-82M - TTS model
• Supertonic-3 - Multilingual TTS model
• espeak-ng - Phonemization engine
• ONNX Runtime - Inference engine

📞 Support

• Documentation: docs/
• Issues: GitHub Issues
• API Docs: http://localhost:5768/swagger (when running)
• Configure the timeout via the MODEL_IDLE_TIMEOUT_SECONDS environment variable
• Set to 0 to disable automatic release and keep all loaded models in memory

Tests

Local Testing

Unit tests:

dotnet test tests/SharpAudio.Api.Tests/SharpAudio.Api.Tests.csproj

Integration tests with Docker:

# Run all tests (unit + integration)
./tests/run-tests.sh all

# Run specific test types
./tests/run-tests.sh unit
./tests/run-tests.sh integration

Or use Docker Compose directly:

docker compose -f docker-compose.test.yml up --abort-on-container-exit

Continuous Integration

GitHub Actions automatically runs tests on all pull requests:

• Unit Tests: Fast, isolated tests (~2 minutes)
• Integration Tests: Full API tests with model downloads (~5-15 minutes)
• Test Reports: Automated test summaries in PR checks

The CI workflow:

• Runs on PRs to main, master, or develop branches
• Caches model files (~1-2GB) to speed up subsequent runs
• Uploads test results and API logs as artifacts
• Reports test failures directly in PR checks

See .github/workflows/README.md for details.

Architecture

HTTP Request → KokoroTtsSynthesizer
                    ↓
              KokoroTtsEngine
                    ↓
        ┌───────────┴───────────┐
        ↓                       ↓
  EspeakWrapper          OnnxRuntime
  (phonemization)        (inference)
        ↓                       ↓
  libespeak-ng.so.1      model.onnx
        └───────────┬───────────┘
                    ↓
            PCM16 WAV Output

ASR follows an analogous path (AsrTranscriberRouter → WhisperAsrTranscriber/ NemotronAsrTranscriber → WhisperAsrEngine/NemotronAsrEngine), running in its own worker process when SERVER_MODE enables ASR. See Architecture for the full diagram covering both TTS and ASR.

Requirements

• .NET 10.0 SDK (preview)
• Docker with Compose v2
• Node.js 24+ with pnpm (for frontend development)
• espeak-ng library (included in Docker image, TTS only)