Sona

On-device voice AI + RAG for macOS. Ask questions about your documents by voice — 100% local.

Features

• Voice in/out — Parakeet TDT STT (1.9% WER) + Piper/Kokoro TTS
• Document RAG — upload PDF, DOCX, XLSX, PPTX, CSV, HTML, TXT and ask questions grounded in your files
• Local LLM — 4-bit quantized models (Llama 3.2, Qwen3/3.5, Liquid LFM2/2.5)
• GPU acceleration — MetalRT on Apple M3+ (~486 tokens/s)
• Web UI — browser interface at localhost:8080
• Shareable — expose via Cloudflare Tunnel with built-in auth

Everything runs on your machine. No API keys, no cloud, no data leaves the device.

Quick start

# Prerequisites
xcode-select --install
brew install cmake

# Build
git clone https://github.com/user/sona.git && cd sona
mkdir build && cd build
cmake .. -DCMAKE_BUILD_TYPE=Release
cmake --build . -j$(sysctl -n hw.ncpu)

# Download AI models (~1GB, first time only)
./sona setup

# Start
./sona
# Open http://localhost:8080

Commands

Options

--port <n>          Server port (default: 8080)
--gpu-layers <n>    GPU layers for LLM (default: 99 = all)
--models <dir>      Models directory (default: ~/Library/Sona/models)
--verbose, -v       Debug logs

API

All POST endpoints require X-Sona-Token header when accessed remotely. The token is printed to stderr on startup and auto-injected into the web UI.

Remote access

Share your instance with others via Cloudflare Tunnel:

brew install cloudflared
cloudflared tunnel --url http://localhost:8080 --protocol http2

The generated https://xxx.trycloudflare.com link includes auth — send it to anyone.

Models

LLM

STT

TTS

Architecture

                         ┌─────────────────────────────────┐
                         │          Web UI / API            │
                         │    localhost:8080 (cpp-httplib)   │
                         └──────────┬──────────┬────────────┘
                                    │          │
                            voice   │          │  document
                            query   │          │  upload
                                    ▼          ▼
┌───────────────────────────────────────┐  ┌──────────────────────────┐
│         Voice Pipeline                │  │     RAG Pipeline         │
│                                       │  │                          │
│  ┌─────┐   ┌─────┐   ┌─────────┐     │  │  ┌────────┐  ┌───────┐  │
│  │ VAD ├──▶│ STT ├──▶│   LLM   │     │  │  │ Doc    ├─▶│Embed  │  │
│  └─────┘   └─────┘   │         │◀────┼──┼──│Processor│  │Engine │  │
│                       │ llama.cpp│     │  │  └────────┘  └──┬────┘  │
│  ┌─────┐              │    or   │     │  │               ┌──▼────┐  │
│  │ TTS │◀─────────────│ MetalRT │     │  │  ┌────────┐  │Vector │  │
│  └──┬──┘              └─────────┘     │  │  │ BM25   │  │Index  │  │
│     │                                 │  │  └───┬────┘  └──┬────┘  │
└─────┼─────────────────────────────────┘  │      │    RRF   │       │
      ▼                                    │      └────┬─────┘       │
 ┌─────────┐                               │           ▼             │
 │CoreAudio│                               │      top-K chunks       │
 │  Output  │                               └──────────────────────────┘
 └─────────┘

 Engines: sherpa-onnx (STT/TTS/VAD) · llama.cpp (LLM) · MetalRT (M3+ GPU)
 Storage: USearch HNSW (vectors) · BM25 (term index) · mmap'd chunks

src/
  api/        C API — public interface for embedding Sona
  audio/      CoreAudio I/O, WAV encoding
  cli/        Web server, embedded frontend, CLI commands
  core/       Lock-free ring buffer, types, config
  engines/    STT, TTS, LLM, embedding, MetalRT backends
  models/     Model registry, download, priority-based auto-selection
  pipeline/   Voice pipeline orchestrator (STT → LLM → TTS)
  rag/        Hybrid retrieval: USearch HNSW + BM25 + RRF fusion

Dependencies: llama.cpp (LLM + Metal GPU), sherpa-onnx (STT/TTS/VAD), USearch (vector index), cpp-httplib (HTTP server)

License

MIT