Sonata

Real-time voice intelligence for Apple Silicon. Native C + Rust. Full-duplex conversation.

Quick Start

brew install curl opus onnxruntime espeak-ng
make
ANTHROPIC_API_KEY=sk-ant-... ./sonata

Why Sonata?

Most voice pipelines are Python glue connecting cloud APIs. Sonata is different — every audio processing component from speech recognition to text-to-speech runs natively on Apple Silicon, using all four hardware compute units simultaneously (GPU + AMX + ANE + NEON). The result: low-latency full-duplex conversation with barge-in support, entirely on your Mac.

Mic → STT → LLM → TTS → Speaker

Note: The default LLM backend is Claude (cloud API). Use --llm local for fully on-device inference with Llama 3.2.

Key Features

• 100% native audio pipeline: C + Rust. Single make builds 44 shared libraries, 6 Rust crates, custom Metal kernels, and one binary.
• Full-duplex barge-in: CoreAudio VoiceProcessingIO with hardware AEC. Speak while the assistant is talking — playback interrupts immediately.
• Fused 3-signal end-of-utterance: Energy VAD + LSTM endpointer + ASR-inline EOU token, weighted and fused for <240ms turn detection.
• Speculative prefill: LLM request fires at 70% EOU confidence. Saves 100-300ms when the prediction is correct.
• Multi-engine STT: Conformer CTC (0.9% WER on LibriSpeech), Kyutai Rust 1B, BNNS/ANE-accelerated. Switch with --stt-engine.
• Apple Silicon optimized: Every stage runs on the right compute unit — Metal GPU for transformers, AMX for DSP, ANE for power-efficient inference, NEON for SIMD.

TTS Status

Sonata currently ships with two production-ready TTS engines and one in-development custom TTS:

The default pipeline uses Piper for TTS. Sonata TTS is a custom from-scratch system designed for Apple Silicon — the inference pipeline is complete and mechanically correct, but the models require training on paired speech data before producing quality output.

Sonata TTS Architecture (In Development)

Text → Semantic LM (241M, Metal GPU) → Semantic Tokens (50 Hz)
     → Flow Matching (36M, Metal/ANE) → Acoustic Latents
     → iSTFT Decoder (5M, vDSP/AMX)  → Waveform (24kHz)

Design innovations:

• Single LM pass per frame — no sequential codebook prediction (unlike Mimi's DepFormer)
• FSQ quantization — 4096-entry codebook with zero codebook collapse
• Conditional Flow Matching — continuous acoustic latents, zero quantization error
• iSTFT decoder — 100x faster than ConvTranspose, AMX-native

Performance (Measured)

Note: Latency numbers are for audio processing stages. End-to-end latency including LLM response time depends on backend (Claude API ~300ms TTFT, on-device Llama ~100ms TTFT).

Architecture

┌───────────────────────────────────────────────────────────────────────────┐
│                              sonata                                       │
│                                                                           │
│  ┌──────────────┐   ┌────────────┐   ┌────────────┐   ┌───────────────┐ │
│  │ CoreAudio     │   │ STT Engine │   │ LLM Engine │   │ TTS Engine    │ │
│  │ VoiceProc IO  │──▶│ Rust/C/ANE│──▶│Claude/Gemini──▶│ Piper/Sonata │ │
│  │ (48kHz, AEC)  │   │ fp32/16/8 │   │  /Local    │   │              │ │
│  └──────┬────────┘   └─────┬──────┘   └────────────┘   └──────┬────────┘ │
│         │                  │                                    │         │
│         │           ┌──────▼─────────────────────────┐         │         │
│         │           │  Speech Detector (unified)      │         │         │
│         │           │  Native VAD + LSTM + ASR Token   │         │         │
│         │           │  → Speculative Prefill (70%)     │         │         │
│         │           └────────────────────────────────┘         │         │
│         │                                                       │         │
│         │           Audio Post-Processing (AMX/vDSP)            │         │
│         │    ┌──────────────────────────────────────────┐       │         │
│         │    │ vDSP Prosody (pitch/volume/EQ/limiter)   │       │         │
│         │    │ Noise Gate (spectral, vDSP FFT)          │◀──────┘         │
│         │    │ LUFS Loudness Normalization (BS.1770)     │                 │
│         │    │ AudioConverter HW Resampler (24→48kHz)   │                 │
│         │    │ Breath Synthesis (Voss-McCartney pink)    │                 │
│         │    │ Spatial Audio HRTF (optional 3D)          │                 │
│         ◀────│ SPMC Ring → Speaker + Opus Encoder        │                 │
│              └──────────────────────────────────────────┘                 │
│                                                                           │
│  ┌───────────────────────────────────────────────────────────────────┐   │
│  │  Conversation Intelligence                                         │   │
│  │  Backchannel · Emotion · Diarizer · Memory · Prosody Prediction   │   │
│  └───────────────────────────────────────────────────────────────────┘   │
│                                                                           │
│  ┌───────────────────────────────────────────────────────────────────┐   │
│  │  Network APIs: HTTP REST · WebSocket · Web Remote                  │   │
│  └───────────────────────────────────────────────────────────────────┘   │
└───────────────────────────────────────────────────────────────────────────┘

Pipeline State Machine

Listening → Recording → Processing → Streaming → Speaking → Listening
                 │           │            ↑
                 │           │  (speculative prefill at 70% EOU)
                 │           │
                 └───────────┘  Barge-in (any state) ──→ Listening

Native Libraries (44 C shared libraries + 6 Rust cdylibs + 1 metallib)

Core Audio Engine

Speech Recognition

Speech Synthesis

LLM Backends

Voice Activity & End-of-Utterance Detection

Audio Post-Processing

Prosody & Expression

Text Processing

Conversation Intelligence

Network & API

Quality & Profiling

Infrastructure

Optimizations

Hardware Utilization

Sonata runs on all four Apple Silicon compute units simultaneously:

Key Performance Techniques

• Lock-free everywhere: SPSC and SPMC ring buffers with 128-byte cache-line aligned atomics. Zero locks in the audio hot path.
• Zero allocation in hot loops: Arena allocator provides per-turn memory. CoreAudio callback does zero malloc/free.
• INT8 quantization: Per-channel symmetric quantization with NEON-vectorized dequantize → cblas_sgemm. ~4x memory reduction.
• Fused operations: vDSP pitch→EQ→volume→limit chain. Interleaved KV cache eliminates separate K/V fetches.
• VM-mirrored ring buffers: mach_vm_remap creates a virtual memory mirror — reads/writes never need to handle wraparound.
• Hardware resampling: Apple AudioConverter for sample rate conversion is higher quality and faster than FIR for non-integer ratios.
• Speculative prefill: LLM API request starts at 70% EOU confidence, saving 100-300ms when the prediction is correct.
• Conformer cache-aware streaming: Per-layer K/V projection and convolution state caching for true frame-by-frame inference without recomputation.
• Custom Metal kernels: Flash Attention v2, fused SiLU+gate, layer norm — loaded at runtime from compiled .metallib.
• Concurrent compute unit dispatch: GPU runs Sonata LM while ANE runs ConvNeXt decoder while AMX runs iSTFT — all three stages overlap.
• Apple perf primitives: RT thread scheduling, huge page model loading, IOSurface zero-copy GPU↔CPU sharing.

Quality Assurance Framework

Sonata includes a comprehensive native C quality benchmark suite for proving STT and TTS quality.

Metrics Implemented

Round-Trip Testing

The ultimate quality proof: text → TTS → audio → STT → transcript → WER(original, transcript). If the round-trip WER is low, both TTS and STT are working correctly.

make bench-quality    # Run benchmark self-tests
make test-quality     # Run all quality metric tests
make test-roundtrip   # Round-trip with mock TTS/STT callbacks

Test Suite

make test    # Run all 30 test suites

Additional targets (not in make test):

make test-sonata-quality     # End-to-end TTS→STT quality with Sonata
make test-sonata-stt         # Sonata STT + refiner API tests
make test-native-vad         # Pure C VAD unit tests
make test-speech-detector    # Unified speech detector tests
make test-apple-perf         # Apple Silicon perf primitives
make test-quality-improvements  # Noise gate + LUFS + voice quality
make bench-vad               # VAD throughput benchmark
make bench-sonata            # Sonata TTS throughput benchmark
make bench-live              # Live model benchmark with real audio
make bench-industry          # Industry-standard quality benchmarks

Additional Features

• INT8 quantization: Per-channel symmetric INT8 weights with NEON-vectorized dequantization. ~4x smaller models with <1% accuracy loss.
• Apple Neural Engine: BNNS Graph-accelerated Conformer encoder and ConvNeXt decoder dispatch to the ANE on macOS 15+ for power-efficient inference.
• Custom Metal kernels: Flash Attention v2, fused SiLU+gate, layer norm, and fp16 GEMM compiled as .metallib and loaded at runtime.
• Per-turn latency profiling: --profiler flag enables nanosecond-precision breakdown of every pipeline stage (STT, LLM TTFT, TTS TTFS, E2E) with P50/P95/P99 statistics.
• Voice quality metrics: PESQ-lite, STOI-lite, Log-Spectral Distance, and MOS prediction for automated TTS quality evaluation.
• Speaker diarization: ONNX-based speaker embedding extraction with cosine similarity tracking for multi-speaker conversations.
• Active listening: Backchannel generation ("mhm", "yeah") from acoustic cues at ~50ms latency — no LLM round-trip required.
• Web remote: Phone browser captures mic via Web Audio API → streams PCM over WebSocket → pipeline processes and streams audio back.
• REST API: HTTP server with WebSocket streaming for integration with external applications.
• Multi-LLM backend: Supports Claude (SSE), Gemini, and on-device Llama 3.2 (1B-3B). Switch with --llm claude|gemini|local.
• Multi-STT backend: Kyutai Rust (1B), pure C Conformer CTC (fp32/fp16/int8), Sonata CTC (RoPE conformer), and BNNS/ANE-accelerated Conformer. Switch with --stt-engine rust|conformer|bnns.

Requirements

• macOS 14+ on Apple Silicon (M1/M2/M3/M4)
• Xcode Command Line Tools (xcode-select --install)
• Rust (curl --proto '=https' --tlsv1.2 -sSf https://sh.rustup.rs | sh)
• Homebrew dependencies: brew install curl opus onnxruntime espeak-ng
• Anthropic API key (ANTHROPIC_API_KEY) for Claude backend (or use --llm local for on-device)

Build

make          # Build everything: 44 C shared libraries + 6 Rust cdylibs + 1 metallib + binary
make libs     # Build just the C shared libraries + metallib
make clean    # Remove all build artifacts

Build output:

1. build/*.dylib — 44 C shared libraries
2. build/tensor_ops.metallib — Custom Metal kernels (Flash Attention v2, fused SiLU+gate, layer norm)
3. src/stt/target/release/libpocket_stt.dylib — Rust STT cdylib (Kyutai 1B, candle + Metal)
4. src/llm/target/release/libpocket_llm.dylib — Rust local LLM cdylib (Llama 3.2, candle + Metal)
5. src/local_llm/target/release/liblocal_llm.dylib — Rust local LLM alternative cdylib (candle + Metal)
6. src/sonata_lm/target/release/libsonata_lm.dylib — Rust Sonata LM cdylib (241M semantic model)
7. src/sonata_flow/target/release/libsonata_flow.dylib — Rust Sonata Flow cdylib (35.7M flow matching)
8. src/sonata_storm/target/release/libsonata_storm.dylib — Rust Sonata Storm cdylib (parallel TTS)
9. sonata — Pipeline binary linking everything

Run

ANTHROPIC_API_KEY=sk-ant-... ./sonata

Speak naturally. The pipeline detects speech onset, transcribes in real-time, sends to the LLM, and streams TTS audio back — all with full-duplex barge-in support.

Options

sonata [OPTIONS]

Voice & Model:
  --voice PATH       Voice .wav or .safetensors path
  --stt-repo REPO    STT HuggingFace repo (default: kyutai/stt-1b-en_fr-candle)
  --tts-repo REPO    TTS HuggingFace repo (default: kyutai/tts-1.6b-en_fr)
  --n-q N            Audio codebooks for TTS (default: 24)

STT Engine:
  --stt-engine E     STT engine: rust (default), conformer, or bnns
  --cstt-model PATH  Conformer STT .cstt model file path
  --bnns-model PATH  BNNS .mlmodelc path (ANE accelerated)

TTS Engine:
  --tts-engine E     TTS engine: rust (default), c (Kyutai C), or pocket (100M)

LLM Backend:
  --llm ENGINE       LLM backend: claude (default), gemini, or local
  --llm-model M      LLM model name (auto-detected per engine)
  --system PROMPT    System prompt

VAD:
  --no-vad           Disable semantic VAD (use energy VAD only)
  --vad-threshold F  Semantic VAD threshold (default: 0.7)

Audio Post-Processing:
  --pitch F          Pitch multiplier (1.0 = normal, 1.2 = higher)
  --volume F         Volume in dB (0.0 = normal, 6.0 = louder)
  --no-hw-resample   Disable AudioConverter (use FIR fallback)
  --spatial AZ       Enable 3D spatial audio at azimuth AZ degrees

Advanced:
  --metallib PATH    Custom .metallib path for GPU kernels
  --profiler         Enable per-turn latency profiling
  --prosody          Enable SSML-aware system prompt

Examples

# Default — conversational voice assistant
ANTHROPIC_API_KEY=sk-... ./sonata

# Custom voice with pitched-up output
./sonata --voice /path/to/voice.wav --pitch 1.15

# Spatial audio — voice positioned 30° to the right
./sonata --spatial 30

# On-device LLM (no API key needed)
./sonata --llm local --llm-model meta-llama/Llama-3.2-1B-Instruct

# Conformer STT with INT8 model
./sonata --stt-engine conformer --cstt-model models/parakeet_0.6b_int8.cstt

# BNNS/ANE accelerated STT (macOS 15+)
./sonata --stt-engine bnns --cstt-model models/parakeet_0.6b.cstt \
  --bnns-model models/conformer_ctc_0.6b.mlmodelc

# Latency profiling
./sonata --profiler

# Custom Metal kernels
./sonata --metallib build/tensor_ops.metallib

# Custom Claude model with system prompt
./sonata --llm-model claude-sonnet-4-20250514 \
  --system "You are a helpful coding assistant."

Project Structure

sonata/
├── Makefile              # Build system (C libs + Rust + binary + tests)
├── README.md             # This file
├── CONTRIBUTING.md       # Contributing guide
├── AGENTS.md             # AI agent guidance
├── LICENSE               # MIT
├── src/                  # 45 C source files + 6 Rust crates + Metal kernels
├── include/              # C header files
├── tests/                # 30+ test suites
├── scripts/              # Benchmarking, conversion, and export scripts
├── web/                  # Web dashboard and remote UI
├── docs/                 # API reference, architecture, troubleshooting
├── examples/             # curl, WebSocket, and integration examples
├── models/               # Downloaded/compiled models (gitignored)
└── build/                # Compiled output (gitignored)

Contributing

See CONTRIBUTING.md for development setup, code style, and pull request process.

Companion Project

pocket-tts — Python/MLX text-to-speech library with custom Metal kernels, Apple AMX/Accelerate DSP, speculative decoding, and OpenAI-compatible API. Sonata uses Rust inference crates (candle + Metal) instead of MLX, but shares the same philosophy of deep Apple Silicon optimization.

License

MIT — Copyright (c) 2026 Seth Ford