esp32-thinks

A multi-model Muse — a personal AI assistant on ESP32, talking out of the box.

Think of it as a Muse that thinks with your models: say "Hi, ESP" (or press the knob), ask anything, and it answers out loud — streaming sentence-by-sentence while the model is still writing, in English or Chinese, on a $50 gadget with a mic, a speaker and a round display.

No vendor lock on the brain: speech, transcription and chat are three independent endpoints, each pointed at any OpenAI-compatible server — one provider for all three, or a different one per task.

• ESP32-S3 · built on Meta's open-source muse-gadget-sdk · MIT
• Models: OpenAI, QuickSilver, or any mix — presets for the common ones, fully custom for the rest

Why

Cloud assistants rent you a brain: one vendor's models, one vendor's pricing, one vendor's uptime, and your voice on their servers. The gadget hardware is fine — the lock is in the firmware.

esp32-thinks flips that. The device runs your stack: pick the cheapest transcription, the smartest chat model, the voice you like — per task, changeable in one config line and a rebuild. The whole pipeline is ~1,600 lines of patch against a stock open-source SDK, so it stays yours to read, tweak, or point at a local server.

What you get

• Wake word + button — say "Hi, ESP" (esp-sr WakeNet10, runs on-chip) or hold the knob; both start a turn. Wake turns end on a voice-activity timer; the button works exactly like before.
• Bring your own brain — three independent OpenAI-compatible endpoints (STT / chat / TTS) in one config file. Mix providers freely; presets: openai, quicksilver, custom.
• Streaming voice replies — the chat stream is parsed at sentence boundaries (ASCII + CJK); each sentence is synthesized the moment it lands and appended to the audio already playing. First word arrives while the model is still generating.
• Warm and fast — persistent TLS sessions; a warm voice turn is ≈ 5–7 s end-to-end, text caption at ~2–3 s.
• Language-aware voice — Chinese turns can use a different (better) TTS model than English ones, picked automatically from the transcript.
• Headless provisioning — put your Wi-Fi in the config; the device self-provisions on boot. No app, no BLE dance, no vendor account needed.
• Silence-trimmed uploads — recordings are trimmed to speech before upload, shaving ~0.5 s off transcription.
• Typed turns speak too — the serial console path gets the same treatment (useful for testing without talking).

At a glance

Quick start

# 0) One-time: ESP-IDF v6.x for esp32s3 (docs/REFERENCE.md §Prerequisites)

git clone https://github.com/facebookincubator/muse-gadget-sdk.git
git clone https://github.com/raullenchai/esp32-thinks.git
cd esp32-thinks

# 1) Back up the factory NVS once — your restore path if anything goes sideways
./scripts/backup-nvs.sh ../muse-gadget-sdk /dev/ttyACM1

# 2) Apply the patch + configure your brain (pick a preset)
./apply.sh ../muse-gadget-sdk --preset quicksilver   # or openai / custom

# 3) Build + flash (CH342-safe pacing, ~2 min)
./build-and-flash.sh ../muse-gadget-sdk /dev/ttyACM1

With a preset + headless Wi-Fi in step 2, that's the whole setup: unplug, re-plug, and talk — press the knob, or just say "Hi, ESP".

Built on muse-gadget-sdk

Credit where it's due — and clarity about the split:

Theirs (upstream, untouched behavior): the device framework, LVGL UI and round-LCD drivers, audio stack (codec, resampler, minimp3), input/console plumbing, OTA and partitioning. This recipe targets the SDK at a pinned commit so patches apply cleanly.

Ours (patches/muse-brain.patch, ~1,600 lines): the brain session (muse_chat_session.cpp — three-endpoint config, SSE sentence streaming, TTS pipeline, keep-alive TLS), the wake-word integration (muse_voice.c — direct WakeNet10 on idle chunks, wake→PTT synthesis, VAD end-of-turn, cooldown), the model flash partition and the esp-dl kernel fix, plus the tooling (apply.sh, build-and-flash.sh, NVS backup, probes).

Nothing else from the Muse cloud is required: the vendor agent works as an optional fallback (its token is optional), and with headless provisioning the device never contacts it.

Configuration

Everything lives in esp32/components/muse/provider_config.h (written by apply.sh):

Changed a define? Re-run ./build-and-flash.sh — it rebuilds and reflashes.

How a turn flows

1. Wake or press → "Hi, ESP" (or the knob) starts recording; the pre-buffer already holds the last second of audio. A wake turn ends after ~1.4 s of silence; a button turn ends on release.
2. You stop talking → the recording is trimmed to speech and POSTed as multipart WAV to the transcription endpoint (keep-alive).
3. The transcript opens an SSE chat stream → the parser splits the reply at sentence boundaries (.!? and CJK 。！？…), so Chinese works too.
4. Each sentence POSTs to the speech endpoint → the MP3 is appended to the audio buffer already being decoded and played. A reserved message slot lets captions and speech interleave with the stream.
5. Both TLS connections stay open for the next turn; the first turn after boot pays one handshake each (~1.5 s), the rest pay none.

Latency receipts (warm, measured on device, QuickSilver preset): transcription 3.4 s, first spoken word ≈ 5–7 s after you stop; the text caption shows at ~2–3 s. Cold adds ~1.5 s. See REFERENCE for the full breakdown and every gotcha.

Security notes

• API keys are compiled into the firmware. This is a hobby recipe, not a multi-tenant product — don't flash keys you aren't comfortable with into a device that leaves your bench, and don't commit provider_config.h (it's generated; apply.sh keeps it out of the repo).
• Each endpoint pins its root CA (files/*.pem); the ESP cert bundle alone doesn't carry them (the v0.1 story — see REFERENCE — now with a second root for Let's Encrypt hosts).
• Voice is recorded only during a turn (button held, or after the wake word until the VAD closes it). With a headless token placeholder the Muse cloud is never contacted.

Roadmap

• More boards the SDK supports (Waveshare, M5Stack…).
• Full-duplex speech-to-speech models (e.g. QuickSilver's qwen3.8-omni) and local LLMs on your LAN — the brain is just three URLs.
• Custom wake phrases ("Hey Mica") via Espressif's TTS Pipeline V3.

Full story, exact code paths and every gotcha found the hard way: docs/REFERENCE.md.

Troubleshooting

Top three (full checklist in REFERENCE):

• No reply at all? Check the console for brain heard: (transcription worked) and TTS ok: — a BRAIN STT FAILED in a quiet room is normal (whisper returns empty); in a real conversation it means the key/URL is wrong.
• device reports readiness to read but returned no data — two processes fighting over /dev/ttyACM1 (monitor + probe). One at a time.
• Garbled flash / boot loop — you used plain esptool at high baud. The CH342 bridge drops whole packets; the recipe uses paced_esptool at 115200 for a reason.