Aokie

Aokie Receptionist Kit — a Windows app + certified USB Bluetooth dongle that turns your existing phone into an AI-answered reception line. Pair your phone to Aokie as a Bluetooth hands-free device using the bundled dongle (see the compatibility matrix below — Aokie talks WinUSB directly to specific chipsets, not the generic Windows Bluetooth stack); incoming cellular calls route their audio to the PC over Bluetooth HFP, and Aokie answers them with a local LLM + TTS pipeline. Aokie also handles SMS over Bluetooth MAP, greets known callers by name via PBAP, and can drive an appointment-booking and ordering flow from inside the call. By default everything runs on-device — no cloud, no API keys, no companion app — but the AI providers are pluggable so you can swap in an OpenAI-compatible HTTP endpoint or run llama.cpp as a sidecar if you'd rather.

Scope in one line: Windows PC + certified Bluetooth dongle + one paired phone + one call at a time. Built for sole-trader and small- business reception flows. Multi-line / call-queue / multi-device is deliberately out of scope.

One call at a time. Aokie pairs to a single phone over HFP and serves one in-progress call. There's one SCO link per active call, one transcription stream, one LLM turn loop. A second incoming call while the first is live goes to the phone's voicemail (or whatever the carrier's busy-signal behaviour is). Multi-line / call-queue support isn't on the roadmap — Aokie is built for sole-trader and small-business reception flows where missing a second simultaneous call is preferable to a half-handled one.

Platform support: Windows is the supported target. Linux is a developer preview — the libusb transport, udev rule renderer, and diagnostic commands work end-to-end in the Rust stack, but packaging, pkexec udev install, and end-to-end SCO audio on real Linux hardware are still outstanding. Run on Windows for any production use; treat Linux Aokie as a contributor-only build until that work lands.

How it works

1. Plug a supported USB Bluetooth dongle into the PC. The compatibility matrix below tracks what's been verified vs. best-effort.
2. From inside Aokie, install the WinUSB driver on the dongle with one click.
3. Pair your phone to "Aokie AI Assistant" as a headset / hands-free device.
4. When a call comes in, Aokie auto-answers, greets the caller (by name if it's in the phone book), transcribes each turn with Whisper, generates a reply with Gemma 4, and speaks it back via Pocket-TTS — all on-device.
5. New SMS messages stream in live and the AI auto-replies if you've enabled it. Booking and order tools land their results into SQLite so the dashboard reflects everything in real time.

Every turn, every SMS, every booking, and every order is logged to SQLite so the History page and the Messages / Appointments / Orders pages all show live state.

Hardware compatibility

The Bluetooth stack is dongle-aware: HFP codec selection, SCO transfer sizing, and PCM endpoint quirks vary by chip family. We test against what we can verify, and grade other dongles by what's known to work versus what's known to misbehave.

If your dongle isn't listed, Aokie will probably enumerate and run, but HFP audio quality, MAP/PBAP timing, and SCO error rates are not guaranteed. The Pairing page's diagnostic runtime button captures the controller report and event stream so a debug session can confirm end-to-end connectivity before going live.

Known limitations

RCS messages won't reach Aokie — disable RCS or accept SMS-only on the paired phone. Bluetooth MAP only carries SMS/MMS; there is no Bluetooth-side path for RCS (Pixel-to-Pixel chat, Google Messages "chat features", Apple's iMessage-style RCS). When RCS is enabled on the paired phone, the carrier may auto-upgrade an SMS thread to RCS without warning, after which Aokie will see the MNS session open and then go dead silent — new messages just never arrive. On Pixel / Android: open Google Messages → Settings → RCS chats and turn it off, or pair a phone whose default messaging app is SMS-only. This is a platform limitation, not an Aokie bug; the only fix is for one of the OS / carrier / Bluetooth-MAP spec stakeholders to add a path, which currently does not exist. See docs/TROUBLESHOOTING.md for the full diagnosis flow.

Other limitations worth knowing up-front:

• One paired phone, one in-progress call. A second incoming call while the first is live goes to the phone's voicemail. See the banner at the top of this README.
• Built-in laptop Bluetooth is unsupported — Aokie binds the dongle exclusively to WinUSB, which would take Windows's own Bluetooth stack offline for the rest of the system. Use a dedicated USB dongle from the certified list above.
• Linux is developer-preview. Pairing UX, packaging, and SCO audio validation aren't done. Run on Windows for any production use.

What's inside

• UI — Tauri 2.10 shell wrapping a React + Vite + Zustand frontend. Pages: Dashboard, Pairing, Settings, AI Stack, Messages, Appointments, Orders, History, Call Detail. App-wide toast surface for backend errors that happen off the request thread.
• Bluetooth — aokie_radio: a from-scratch Rust HCI / L2CAP / RFCOMM / SCO / SDP / OBEX / HFP / MAP / PBAP stack that talks to the dongle directly over WinUSB. No vendored C, no FFI shim. Includes a pairing store, eSCO with mSBC wide-band + CVSD fallback, and a cargo run --bin aokie-radio-diag tool for bring-up (Windows only — the binary exit(1)s on other platforms; Linux uses the Tauri-side diagnostic commands instead). SCO iso submission, voice setting, alt selection, and Broadcom vendor routing mirror BTstack's USB transport (hci.c + hci_transport_h2_winusb.c) — see ISSUES.md for the audit history if you need to add support for a new chipset.
• USB dongle setup — aokie_dongle: Rust SetupAPI enumeration plus a generated WinUSB INF that's installed by aokie-driver-helper (an elevated sidecar pointed at by the AOKIE_WINUSB_HELPER env var).
• AI provider registry — ai::{LlmProvider, TtsProvider, SttProvider} traits with adapters for the catalogue below. Active provider is picked per-surface from <app_data>/ai_providers.json and is hot-swappable at runtime via Settings → AI Providers.
• STT (speech recognition) — pluggable across:
  • Candle Whisper — Whisper large-v3-turbo (Hugging Face safetensors) loaded via candle. Hand-rolled rustfft mel front-end matching torch.stft(center=True).
  • Sherpa-onnx Whisper — sherpa-onnx-packaged Whisper bundles (encoder + decoder + tokens). Lighter footprint than Candle, same Whisper accuracy.
  • Parakeet-Unified-EN-0.6B — NVIDIA FastConformer + RNN-T, int8 ONNX. Eager-loaded; the receptionist's lowest-latency English path.
  • Moonshine-tiny — UsefulSensors Moonshine encoder-decoder (English by default; tiny-zh / tiny-ja / tiny-ko / tiny-ar / tiny-uk / tiny-vi / base-es language packs available too). ~80 MB on disk and surprisingly good for the size.
  • Qwen3-ASR-0.6B int4 — Alibaba multilingual encoder-decoder with the Qwen3-0.6B chat backbone as decoder. Pinned-language primer pushes it past auto-detect on short / accented English.
• LLM — Gemma 4 multimodal via ONNX Runtime 1.25 (GPU via CUDA when available, CPU fallback otherwise). Streams tokens turn-by-turn. Or pick Qwen 3.5 4B Q4_K_M (GGUF) from the catalogue to route through the bundled llama-server sidecar (text-only), or OpenAI-compatible HTTP for any remote / LM Studio / Ollama endpoint. Provider switch unloads the previous in-process runtime immediately so memory drops on save.
• TTS — Pocket-TTS ONNX (kyutai-derived, voice-cloning capable, pure-Rust ort + rustfft pipeline, supports .safetensors voice states and reference-wav cloning via the mimi_encoder). Or Sherpa-onnx TTS — VITS / Piper / Kokoro voice bundles via sherpa-onnx, broader voice catalogue at the cost of bundle download size.
• VAD — Silero VAD via sherpa-rs for endpoint detection during calls.
• DB — SQLite (typed Rust commands, no JS plugin) storing call logs, per-turn transcripts, SMS threads, contacts, bookings, and orders. All multi-row writes run inside IMMEDIATE transactions.
• PII redaction — phone numbers / message bodies / transcripts mask in logs unless AOKIE_VERBOSE_LOGS=1.

Repo layout

website/                              Marketing / sales site (Vite + React + TS,
│                                     handwritten CSS, editorial aesthetic)
aokie-desktop/
├── src/                              React frontend
│   ├── pages/                        Dashboard · Pairing · Settings · AI Stack ·
│   │                                 Messages · Appointments · Orders · History · Call Detail
│   ├── stores/                       Zustand stores (call / transcript / connection /
│   │                                 config / toast / theme)
│   └── hooks/                        useAppInit, useBluetooth, …
└── src-tauri/                        Rust backend
    └── src/
        ├── ai/                       Provider trait + adapters (LLM / TTS / STT)
        │   ├── adapters/             OnnxGenAi, HttpOpenAi, OnnxTts, SherpaOnnx*, WhisperCandle
        │   └── runtimes/             Heavy modules each adapter wraps
        ├── commands/                 Tauri commands (Bluetooth / USB / ML / call log /
        │                             SMS / calendar / orders / AI config)
        ├── sidecars/                 External-process lifecycle (llama_server)
        ├── bluetooth/                Thin wrapper over aokie_radio
        ├── aokie_radio/              From-scratch HCI/L2CAP/RFCOMM/SCO/SDP/OBEX/HFP/MAP/PBAP
        │   ├── obex.rs, bmessage.rs, vcard.rs
        │   ├── map_*.rs              MAS read, MNS push, MAP listing, runtime driver
        │   └── pbap.rs, pbap_runtime.rs
        ├── aokie_dongle/             USB dongle enumeration + WinUSB INF generator
        ├── bin/aokie-driver-helper.rs  Elevated WinUSB driver-installer helper (validated args)
        ├── bin/aokie-radio-diag.rs     CLI for hardware bring-up + diagnostics
        ├── calendar/                 Bookings + service catalogue
        ├── orders/                   Products + order lifecycle
        ├── msbc/                     Pure-Rust SBC + H2 framing for mSBC SCO
        ├── vad/                      Silero VAD wrapper
        ├── redact.rs                 Phone / Text Display wrappers (PII off by default)
        ├── notepad.rs                Continuous caller transcription buffer
        ├── streaming.rs              Sentence extractor + streaming TTS controller
        └── database/                 SQLite schema + migrations + central open helper

Getting started

Prerequisites

• Windows 10 or 11 (x64)
• Visual Studio 2022 Build Tools with the C++ desktop workload (MSVC 19.4x + Windows 11 SDK)
• Rust stable
• Node.js 20+ and npm
• CPU is the default build target — default = ["custom-protocol", "sherpa"] in aokie-desktop/src-tauri/Cargo.toml. The sherpa feature pulls in sherpa-rs-sys, which CMake-builds espeak-ng + piper-phonemize + cppjieba + openfst from source on first compile (~5 min cold). Stock VS Build Tools handles this on a dev box; if you want a faster bring-up build, opt out with --no-default-features --features custom-protocol — Aokie falls back to stub Sherpa runtimes whose load() returns "feature not compiled in" (the rest of the provider stack — Pocket-TTS, Whisper, Parakeet, Moonshine, Qwen3-ASR, llama-server — keeps working). CUDA isn't needed: Gemma 4 on CPU is slower than GPU but Whisper and Pocket-TTS are fine.
• CUDA 12.8 + cuDNN 9 is opt-in via the cuda cargo feature for GPU-accelerated Gemma 4. See docs/CUDA_SETUP.md for the toolkit layout and the bundle-resource overrides needed for a CUDA release.
• A USB Bluetooth dongle. BCM20702A0 (0a5c:21ec) verified end-to-end with bidirectional mSBC SCO; other WinUSB-capable adapters should work after running the in-app driver installer, but only Broadcom 21ec has been exercised through the full call flow so far.

Build & run (CPU, default)

cd aokie-desktop
npm install
npm run tauri:dev

npm run tauri:dev runs scripts/dev-with-port.mjs, which probes for a free port starting at 5173 and pins Vite + Tauri to it. Plain npm run tauri dev works too but can fail when an old Vite server is still bound to the configured port.

Build & run (CUDA, opt-in)

CUDA isn't in the default feature set, so opt in by passing cargo args through the tauri-cli's -- separator:

cd aokie-desktop
npm install
npm run tauri:dev -- -- --features cuda

For a CUDA release bundle, override bundle.resources at build time so the cuDNN family + provider DLLs ship alongside the exe — see the inline note in aokie-desktop/src-tauri/Cargo.toml next to the cuda feature for the full resource list, and docs/CUDA_SETUP.md for how to populate aokie-desktop/src-tauri/resources/ with the DLLs.

On first launch, open Settings → AI Stack and download the model files. The default catalogue pulls ~7 GB total — Gemma 4 E4B (≈5.5 GB), Whisper (≈1.5 GB), Pocket-TTS (≈250 MB). Plan for ~15-30 minutes on a typical home connection; the AI Stack page reports per-file progress and resumes interrupted downloads on retry. The smaller catalogue entries (Gemma 4 E2B at ≈2.5 GB, Qwen 3.5 4B GGUF at ≈2.5 GB) are listed in the AI Providers picker if you'd rather trade quality for download size on a slow link. Then in Pairing:

1. Pick your Bluetooth dongle from the dropdown.
2. Click Install WinUSB Driver and accept the UAC prompt.
3. Unplug and replug the dongle (Windows needs to re-enumerate it).
4. Click Start Bluetooth Service.
5. On your phone, pair to "Aokie AI Assistant" as a headset.

Place a test call. The caller's audio routes to Aokie instead of the phone's earpiece, the AI greeting plays into the caller's earpiece, and Aokie transcribes each turn live. Post-call extraction lands any bookings or orders the caller arranged into the dashboard.

Configuration

All runtime config (system prompt, greeting, voice selection, auto-answer delay) lives in Settings, persists via Zustand persist, and is mirrored to disk so the Rust backend can read it on startup. The SQLite schema is created automatically on first run under the app data directory.

The active AI providers (LLM / TTS / STT) are picked from the bundled defaults plus an optional override at <app_data>/ai_providers.json. Set the kind field per surface to swap engines:

Edit, save, and the cache hot-reloads — no app restart needed. The startup warm-up only loads the in-process runtime that matches the active kind, so switching to openai-http doesn't waste GPU memory on an unused Gemma load.

The warm-up itself is also opt-out: Settings → AI model warm-up flips between "Fast calls" (preload Whisper / Gemma / Pocket-TTS at launch — first call answers without model-load latency) and "Fast startup" (defer the warm to first inference — the launch-to- dashboard window is shorter and idle memory is smaller, in exchange for a multi-second first-call latency). Each initialize_* command is idempotent, so deferring is safe.

Continuous integration

CI runs in three tiers: fast Linux gates on every push/PR, Windows shape coverage on Windows-relevant push/PR, and dependency audits on a weekly cron + manual dispatch. The split keeps round-trip latency tolerable on JS-only / docs-only changes (Linux Rust + Vite finish in ~3-8 min cached) while still gating Bluetooth / WinUSB / Tauri-config changes on the slower Windows runner.

• frontend-fast (ubuntu-latest) (every push/PR) — ESLint, build-mode tsc -b (catches tsconfig.node.json drift tsc --noEmit would miss), and a Vite production build.
• frontend-windows (windows-latest) (manual / scheduled) — Same checks on a Windows runner; catches Win32-specific frontend regressions (path separators, CRLF, native-dep build errors). Manual because cold runner is 8-12 min and CRLF rarely breaks.
• rust-fast (ubuntu-latest) (every push/PR) — cargo fmt --check + cargo check --lib + cargo test --lib for the cross-platform code (parsers, msbc, MAP/PBAP wire-format, calendar, orders, redact, fuzz harnesses).
• rust-windows (windows-latest) (auto on push/PR touching aokie-desktop/src-tauri/** or this workflow; manual dispatch for ad-hoc runs) — cargo check/test on the full lib and the aokie-driver-helper bin under --no-default-features --features custom-protocol. The sherpa cargo feature is opt-out: with it off, sherpa-rs-sys's CMake build (which windows-latest can't satisfy without ZLIB / ICU dev libs) is skipped and the runtime falls back to stub SherpaOnnxTtsRuntime / SherpaOnnxSttRuntime / StreamingVad impls. That catches Windows regressions in the WinUSB transport, aokie_dongle, the SetupAPI / WinTrust / catalog-sign surfaces, and any windows- sys version drift. Production Windows release builds keep default = ["custom-protocol", "sherpa"] and still pull sherpa-rs in for the full provider catalogue.
• audit-rust (weekly cron + manual) — cargo audit --deny warnings plus cargo deny check. Off the PR critical path so cycles don't page on every transitive-dep advisory.
• audit-js (weekly cron + manual) — npm audit --audit-level=high. Same off-the-PR-path rationale.

No remaining Windows lib-coverage gap — the sherpa feature gate let CI's Windows job compile and test the full lib. The only Windows-specific path still skipped on CI is hardware-in-loop testing (real BT dongle, real SCO audio); that stays a manual / nightly step on the dev box.

aokie-desktop/src-tauri/.cargo/config.toml carries Windows-specific build settings (target-dir = "C:\\t" MAX_PATH workaround + Visual Studio 2022 CMake generator pins). cargo's [env] table is non-forcing, so contributors on Linux / macOS can override CARGO_TARGET_DIR and the CMAKE_GENERATOR* vars in their shell without editing the file.

Project status

End-to-end happy path is verified: pair → answer → bidirectional mSBC audio → Whisper / Gemma / Pocket-TTS turns → MAP message push → post-call booking extraction. See ISSUES.md for the current open list (low-severity at time of writing — Win32 87 bursts during heavy SCO TX, ACL accumulator desync during long MAP messages; both auto-recover and don't block calls).

For the consolidated "what's shippable today, what's pilot-only, what blocks public release" view (signing status, hardware support, CI gates, pre-ship checklist, recording opt-in story), see docs/RELEASE_READINESS.md. It's the single source of truth — other docs should point back to it instead of re-stating release status.

Troubleshooting

• Pairing page stuck on "Detecting…" — usually means the WinUSB driver install hasn't been picked up yet. Unplug + replug the dongle; if it still doesn't show, re-run the driver installer from the Pairing page.
• No audio from the greeting — open AI Stack and confirm all three models show "Ready". On startup Aokie warms whichever in-process runtime matches the active provider kind, so cold models shouldn't be the cause unless the runtime gates file-existence + active-kind rejected the warm. If initialization fails, the log under the app data dir (%APPDATA%/com.aokie.app/ on Windows, ~/.local/share/com.aokie.app/ on Linux) has the details.
• Database write failed toast — backend background DB writes (transcript persists, call_log updates, SMS persistence) emit a db-error event when they fail; the toast is the surface. Common cause is a locked DB (some other process holding the file).
• MAP / PBAP doesn't work — the phone has to grant SMS-over- Bluetooth permission; on Pixel that's a per-bond toggle that appears after the first successful MAS connect, not before. If the toggle is missing entirely, the recovery is a full Bluetooth cache wipe (Settings → System → Reset options → Reset Wi-Fi, mobile & Bluetooth) followed by a re-pair.
• AokieRadio diagnostics — the Pairing page includes a WinUSB HCI controller probe that reads adapter address, version, features, and packet buffers without starting the live call stack. The diagnostic controller initializer applies Aokie's local name, class of device, page timeout, link policy, voice setting, Simple Pairing mode, and connectable / discoverable scan settings. A bounded event listener can capture typed HCI events during pairing experiments and auto-reply to Simple Pairing prompts (no-input/no-output capability, user-confirmation accept, stored link-key replies, negative link-key reply when no key exists, legacy PIN 0000 fallback). The bounded runtime drives ACL / SDP / RFCOMM / HFP responses, accepts SCO/eSCO links, reports incoming PCM stats, auto-answers when AOKIE_RADIO_AUTO_ANSWER=1, writes SCO silence when AOKIE_RADIO_SCO_TX_SILENCE=1, and writes a 440 Hz test tone when AOKIE_RADIO_SCO_TX_TONE=1. Diagnostic link keys persist under the app data directory at aokie_radio/pairing_store.json. The same hardware checks are available from the terminal on Windows with cargo run --bin aokie-radio-diag -- hci-interfaces, probe, init, or runtime 5 from aokie-desktop/src-tauri. The binary refuses to run on Linux / macOS — those platforms have access to the same diagnostics through the in-app aokie_radio_* Tauri commands instead. Set AOKIE_RADIO_DUMP=1 for transport-level hex dumps while testing hardware. Set AOKIE_VERBOSE_LOGS=1 to disable PII redaction in logs (off by default — phone numbers + message bodies + transcripts mask).

Licensing

This project is proprietary to Aokie (aokie.com).

• Pocket-TTS / kyutai voices: see the upstream model cards.
• Whisper and Gemma 4 weights: see their respective Hugging Face cards.