Voice PE Realtime

Turn a Home Assistant Voice PE into the voice assistant you actually wanted — natural speech-to-speech conversation powered by the OpenAI Realtime API, sub-second smart-home control, a wake word trained on your household's voices, an assistant that knows who's speaking, remembers what you tell it, and finds you when long-running work is done. Built on Home Assistant / OpenClaw: Home Assistant runs your home, OpenClaw is its memory, its hands, and its phone.

It runs on your Home Assistant box. Wake-word detection, speaker identity, voice recordings, and memory all stay local. The cloud only hears you after you wake it.

What it feels like

"What's Grandma's number?" Instant recall from your household's long-term memory — answered in under a second. Facts you teach it by voice are built in and stay on your machine; connect OpenClaw and it reaches everything your agent remembers too.

"Research flight prices to London for October." "I'll look into it and report back." It hands the task to OpenClaw, which browses the web in the background for as long as it takes — then the result is announced out loud in the room you asked from, or texted to you if you've stepped out. Long-running tasks that find you when they're done.

Walking into the kitchen: "Set a pasta timer for 9 minutes." Nine minutes later: "Alex, your pasta timer is done" — spoken personally to whoever set it. A gentle bell follows only if nobody responds. Dismiss with a word or the button.

It knows who's speaking. On-device wake word, local voice recognition. It can greet you by name, keep per-person context, and restrict chosen tools to specific speakers — enforced below the model, so it can't be talked around.

"Remember that we park at the north lot." Teach it standing rules by voice. They persist forever, attributed to whoever said them, and only identified household voices can change them. "Forget that" removes them; "what do you remember?" reads them back.

And it just converses. Speech in, speech out — no STT→LLM→TTS chain, so tone and timing feel human. Interrupt it mid-sentence with "stop". Follow up without repeating the wake word. Lights, climate, media, and shopping lists respond in under a second.

What people do with it

Marked † = needs the optional agent integration — built for OpenClaw, works with any agent. Everything else is built in.

• "What's the wifi password?" — say "remember the wifi password is…" once, and it's answered instantly forever. Same for the pool gate code, shoe sizes, where the spare key lives.
• "When's Grandma's birthday?" † — sub-second recall from OpenClaw's long-term memory.
• "What did we decide about the fence contractor?" † — decisions and history, not just facts.
• "Text Sam we're running ten minutes late." † — hands covered in flour; OpenClaw sends it through any of its channels (iMessage, Telegram, WhatsApp, …).
• "Call the pharmacy and ask if my prescription is ready, then tell me what they say." † — pair it with OpenClaw and my openclaw-voice-call-realtime plugin, which gives your assistant a real phone: it places the call, runs the errand, and the answer is spoken back in the room you asked from.
• "Research flights to Tokyo in October and text me the three best options." † — acknowledged now, browsed in the background for as long as it takes, delivered when done.
• "Add everything for lasagna to the shopping list." — native Home Assistant list tools, instant. Then "set a pasta timer" — dismissed or delivered by name when it's done.
• A voice for your automations. † — the announce endpoint accepts any authorized POST, so OpenClaw's scheduled jobs (or any script on your LAN) can speak in the room: "leave in fifteen minutes for the school run."

Longer versions, with the how-it-works behind each: Stories.

Features

• OpenAI Realtime speech-to-speech — gpt-realtime-2 by default, any model id via custom
• Native Home Assistant control via the official MCP Server integration — scoped to exactly the entities you expose
• Custom wake word — "Hey Leonard" ships as the default (trained by this project); switch to Hey Jarvis / Okay Nabu from a dropdown in HA, or train your own
• Speaker recognition — local voice-print identification with guided voice enrollment (say "train my voice")
• Voice-instructed memory — "remember…" / "forget…" / "what do you remember?", speaker-gated writes
• OpenClaw integration — sub-second recall_memory from your agent's memory, deep questions escalated to a full agent turn; a ready-to-run bridge ships in this repo (contracts are agent-agnostic)
• Long-running task delegation — OpenClaw reports back by voice, in the room that asked, via the announce endpoint
• Voice timers — personal announcement → grace period → gentle bell, dismissed by button or voice
• False-wake flagging — by voice, double-press, or automatically; feeds a weekly retrain flywheel
• Web search — current info via a single extra OpenAI call (on by default)
• HA sensors — current speaker, active timers, wakes today, false wakes today, enrollment active
• Persona fully yours — rewrite the instructions; ten OpenAI voices to build on
• Production hardening — proactive session refresh before OpenAI's 60-minute cap, reconnect recovery, echo/ghost-turn guards, stop-word authority, turn-liveness watchdogs

Architecture at a glance

Home Assistant Voice PE           Home Assistant (your box)              Cloud
┌─────────────────────────┐   WS   ┌──────────────────────────┐   WS   ┌──────────────┐
│ custom ESPHome firmware │ ─────▶ │ this add-on              │ ─────▶ │ OpenAI       │
│ wake word + XMOS DSP    │ 16 kHz │ (session, tools, memory, │ 24 kHz │ Realtime API │
│ thin audio client       │ ◀───── │  speaker ID, timers)     │ ◀───── │              │
└─────────────────────────┘        └───────────┬──────────────┘        └──────────────┘
                                               │ tools
                                               ▼
                              HA MCP Server → controls your home
                              OpenClaw    ←→  recall / delegate / announce (optional)

Three parts:

1. Firmware (voicepe-realtime-firmware) — turns the Voice PE into a thin, low-latency audio client. Wake word runs on-device.
2. Backend add-on (this repo) — owns the OpenAI Realtime session, Home Assistant tools, speaker identity, timers, and memory.
3. OpenClaw integration (optional) — deep recall, messaging, calls, and long-running task delegation (agent-agnostic contracts). Everything else works without it.

Quick start

1. Install the add-on: Settings → Add-ons → Add-on Store → ⋮ → Repositories → add https://github.com/TristanBrotherton/voicepe-realtime → install OpenAI Realtime 2 Voice Agent. Set your OpenAI API key.
2. Give it your home: add Home Assistant's MCP Server integration and expose the entities you want voice-controlled to Assist.
3. Flash the firmware: adopt your Voice PE in ESPHome Builder and paste in the device stub from the firmware repo. First flash over USB, updates OTA.
4. Say "Hey Leonard" and ask for a light.

Full walkthrough (~30–45 minutes from zero): Getting Started.

Documentation

The firmware lives in its own repo: TristanBrotherton/voicepe-realtime-firmware.

Credits

• Backend forked from fjfricke/ha-openai-realtime (Felix Fricke).
• Firmware thin-client design based on maxmaxme/home-assistant-voice-pe, a fork of esphome/home-assistant-voice-pe (Nabu Casa / ESPHome).
• Inspiration from marcinnowak79/home-assistant-voice-pe (gemini-live-proxy).
• Built on pipecat-ai, the OpenAI Realtime API, and the official Home Assistant MCP Server integration.

License

MIT. The firmware repo carries the upstream ESPHome license.