openclaw-voice-assistant

Deutsche Version

Wakeword-driven voice assistant for Raspberry Pi. Connects local speech input to OpenClaw as the AI backend and Speaches for GPU-accelerated STT/TTS.

Pipeline

Audio Frontend (ALSA mic  OR  ReSpeaker XVF3800 via ESPHome)
  → openWakeWord ("hey jarvis")
  → WebRTC VAD + recording (max 30 s)
  → STT: [llama.cpp audio model →] [onnx-asr →] Speaches  (fallback: faster-whisper local)
  → Diarization (parallel): Speaches /v1/audio/diarization with known speakers
  → Voice actuator (optional): a switching command? → run locally (~0.5 s), skip the rest
  → Confirmation TTS ("I understood…") — parallel thread
  → POST /v1/responses → OpenClaw  (wrapper: "🎤 [Sprecher: jochen|unbekannt] {text}")
  → Reply TTS sentence by sentence: Speaches /v1/audio/speech  (fallback: Piper local)
  → Mirror query + reply to Telegram

Requirements

• Raspberry Pi (tested: Pi 4/5, ARM64, Raspberry Pi OS Bookworm)
• Python 3.11.9 (exact — openwakeword + tflite-runtime require this version on ARM64)
• OpenClaw running locally on http://127.0.0.1:18789
• Speaches GPU container reachable (default: http://<speaches-host>:8000)
• (optional) voice-analysis container (default: http://<speaches-host>:8001) — adds the acoustic mood signal and the voice_analyze_last_output tool; the assistant runs fine without it.

Mode: local — ALSA microphone + ALSA speaker + optional WLED LED strip

Mode: respeaker — ReSpeaker XVF3800 4-mic array + XIAO ESP32-S3, controlled via ESPHome Native API (aioesphomeapi). No Home Assistant required.

Installation

1. Clone the repository

# Verzeichnisname bewusst mit Unterstrichen: die systemd-Unit erwartet
# %h/openclaw_voice_assist
git clone https://github.com/jochen/openclaw-voice-assistant.git openclaw_voice_assist
cd openclaw_voice_assist

2. Install Python 3.11.9 via pyenv

openwakeword and tflite-runtime are not available for newer Python versions on ARM64. Exactly Python 3.11.9 is required.

curl https://pyenv.run | bash

# Add to ~/.bashrc or ~/.zshrc:
export PYENV_ROOT="$HOME/.pyenv"
export PATH="$PYENV_ROOT/bin:$PATH"
eval "$(pyenv init -)"

sudo apt install -y build-essential libssl-dev zlib1g-dev libbz2-dev \
  libreadline-dev libsqlite3-dev libffi-dev liblzma-dev

pyenv install 3.11.9

The repo includes a .python-version file — pyenv activates 3.11.9 automatically.

3. Create venv and install dependencies

python -m venv ~/openclaw_voice_assist/ow-venv
source ~/openclaw_voice_assist/ow-venv/bin/activate
pip install -r requirements.txt

4. Download openWakeWord models

python -c "
from openwakeword.model import Model
Model(wakeword_models=['hey_jarvis'], inference_framework='tflite')
"

Models are downloaded to /tmp/ow_models_min.

5. Create configuration

cp config.example.yaml config.yaml

Edit config.yaml. Common fields:

Mode: local — additional fields:

Mode: respeaker — additional fields:

6. Download Piper TTS models (local fallback + wakeword ack)

piper-tts is in requirements.txt; the models are gitignored and loaded from hardcoded paths under <project>/models/piper/:

mkdir -p models/piper && cd models/piper
BASE=https://huggingface.co/rhasspy/piper-voices/resolve/main/de/de_DE
wget $BASE/thorsten/low/de_DE-thorsten-low.onnx
wget $BASE/thorsten/low/de_DE-thorsten-low.onnx.json
wget $BASE/thorsten_emotional/medium/de_DE-thorsten_emotional-medium.onnx
wget $BASE/thorsten_emotional/medium/de_DE-thorsten_emotional-medium.onnx.json
cd ../..

Each model needs both the .onnx and its .onnx.json sidecar. If the models already exist elsewhere (e.g. ~/.local/share/piper/), symlinking them into models/piper/ works too.

Running

source ~/openclaw_voice_assist/ow-venv/bin/activate
python -m voice_assistant

The entry point re-execs itself inside the correct venv automatically.

Override profile: GASTON_PROFILE=clawdpi_rs python -m voice_assistant

Autostart (systemd user service)

A unit template ships in systemd/openclaw-voice-assist.service:

mkdir -p ~/.config/systemd/user
cp systemd/openclaw-voice-assist.service ~/.config/systemd/user/
systemctl --user daemon-reload
systemctl --user enable --now openclaw-voice-assist
loginctl enable-linger pi        # start without an active login session

The unit pins PATH to include ow-venv/bin so the service finds piper and aplay for its subprocess calls. Logs:

journalctl _SYSTEMD_USER_UNIT=openclaw-voice-assist.service -f

Profiles

Profile is selected automatically by hostname, or set via GASTON_PROFILE:

ReSpeaker Setup (mode: respeaker)

The ESP32-S3 firmware is in esphome/respeaker.yaml. Flash via:

# Initial (USB):
esphome-venv/bin/esphome run esphome/respeaker.yaml --device /dev/ttyACM0

# OTA:
esphome-venv/bin/esphome run esphome/respeaker.yaml --device respeaker-openclaw.local

The ESPHome venv is separate from ow-venv:

python -m venv esphome-venv
esphome-venv/bin/pip install esphome

How it works: The Pi connects to the ESP via ESPHome Native API (port 6053, aioesphomeapi). Audio streams continuously via the voice_assistant component in API_AUDIO mode. TTS output is sent back as WAV — the Pi serves the WAV over HTTP (port 18800) and the ESP fetches and plays it.

Wakeword detection (openwakeword) runs on the Pi against the audio stream.

Playback goes through the media_player entity, not the voice-assistant announce API — and that is deliberate. The announce API (send_voice_assistant_announcement_*) ends the ESP's voice_assistant session: handle_stop fires, the audio stream breaks off, and the session has to be restarted afterwards. The consequence is easy to miss and hard to debug: the Pi is deaf for the entire duration of every spoken reply. No wake word, no interruption, nothing. Sending the URL to the media_player entity instead (media_player_command(media_url=…, announcement=True)) leaves the session untouched, so the microphone streams straight through playback — and unlike an announcement, playback can be stopped mid-sentence. No firmware change is needed for this; ESPHome's own on_announce performs exactly the same media-player call internally.

Two things to know if you build on this:

• An announcement reports itself as PLAYING, not ANNOUNCING. Waiting for MediaPlayerState.ANNOUNCING to detect playback means waiting for something that never arrives — every sentence then runs into your start timeout. Accept both states (see RespeakerClient._BUSY_STATES). This cost us 5 seconds of added latency per sentence before we noticed.
• Do not make the state event the condition for "playback finished" — make the file length the condition and the state merely an anchor. ESPHome only reports state changes, so if the player stays in a playing state from one sentence to the next, no event arrives at all. Our first attempt waited for that event and ran into its timeout on every such sentence: measured overhangs of +5.4 s and +5.5 s between sentences of one spoken answer, which sounds exactly like the assistant stuttering. Using the WAV length as the primary measure (with a measured ~0.6 s fetch/decode assumption when the anchor is missing) brought it to +0.48…+0.54 s.
• Give every announcement its own URL. Ours reused one filename per worker thread, so all sentences of a reply were served under the same address — one more reason for a player not to report a change.
• Have the sink report an unusual overhang itself. When ours went wrong, the log held only two terse warnings; the gaps had to be computed by hand. A line that fires when playback takes more than ~2 s longer than the audio turns the next regression of this kind into something you read instead of something you hear.

A phone as a satellite (satelliten:, optional)

One microphone doesn't cover the whole home. For rooms you rarely talk in (the bathtub, the garden) you don't need a second ReSpeaker: an Android phone running Ava presents itself on the network as an ESPHome voice satellite and speaks the same Native API. List it as a satellite in the profile:

satelliten:
  - name: phone          # free-form; shows up in the log
    host: 192.168.1.50   # a DHCP reservation helps
    volume: 1.0
    http_port: 18801     # each device needs its own port to fetch replies from

While the app is not running nothing changes, and the assistant keeps quietly retrying the connection. Once it is running, the turn belongs to the device that started it: audio comes from the phone, the reply is played on the phone, and the main microphone is ignored until the turn is over. Timers, speaker recognition, the actuator and the brain stay shared, because it is still one assistant, not two. Later announcements (voice_speak_text, a ringing timer) also go to the phone, until a turn comes from the main microphone again or the phone disconnects. The main device's LED ring follows the turn as usual, which tells the room the assistant is busy.

How Ava differs from the ReSpeaker (measured, not assumed):

• No continuous stream. Ava has no start button. It detects its wake word on the phone (its own models, e.g. "Okay Nabu") and opens a session only then. The session takes the place of the trigger; your own wake word model never hears this audio. Upside: no near-misses on that path.
• The session ends only when the client ends it. Endpointing stays on the assistant's side, which then sends STT_VAD_END + RUN_END.
• Use the announce API for replies, not the media player. On Ava it reports the end of playback (≈0.2–0.5 s after the audio), and with start_conversation the phone opens its microphone by itself afterwards. Follow-ups and clarifying questions run on that, without a wake word.
• Always use dialog endpointing (2 s), never the 1 s command endpointing. The wake word is not part of the recording, so every recording looked "spoken in one breath". Anyone who waited for Ava's wake sound was cut off after a second.
• Recording and playback keep working with the screen locked.
• The brain is told that the input came from the phone (locale.satellit_hinweis), so it can treat a personal phone as its owner, e.g. for a per-person play history. The [Sprecher: …] stays the measured result.

Not done yet: speaker recognition thresholds were measured on the ReSpeaker only, and a phone microphone sounds different. Barge-in has no audio on this path while the phone is speaking.

Lower the music during a turn (musik_ducking:, optional)

If music plays in the room of the main microphone, it can be turned down while a turn runs: from the wake word through the reply, follow-ups and clarifying questions, until the assistant is idle again. The other side is a small REST API of the music system with leases (here: a bridge in front of Lyrion Music Server):

POST {url}/api/v1/rooms/{raum}/duck    {"source", "level"?, "timeout_s"?}
     -> {"ducked": true, "lease_id": "…"}  or  {"ducked": false, "reason": "…"}
POST {url}/api/v1/rooms/{raum}/unduck  {"lease_id"}

musik_ducking:
  enabled: true
  url: http://music-bridge.local:8765
  token_file: ~/music-bridge-token.txt   # Bearer token; not in config.yaml
  raum: livingroom                       # room of the main microphone

The calls run in their own thread and never hold up the voice pipeline; if the bridge is unreachable, this is logged once. The music system restores the volume by itself after timeout_s (30 s) without a new duck, so a longer turn renews every 15 s, but only if the first duck actually lowered something. Otherwise the assistant would duck music it has just started in this very turn. Turns from a satellite (phone) have no room and duck nothing. If a turn started the music (nothing played when the wake word came, something plays now), no follow-up window opens: it would listen straight into the music. Anyone who wants more says the wake word, which ducks at once. Announcements from outside a turn (voice_speak_text) duck the music while they play, unless they go to the phone. Kitchen timers deliberately don't: neither the ringing nor "the timer has expired".

Wake-word level gate (wake_rms_min)

Besides the score gate, there is an optional level gate: the RMS of the loudest 300 ms window in the wake-ring buffer must clear a threshold, or the trigger does not fire — even when the score says yes. Its purpose is to block soft false triggers (TV, keyboard, distant talk) that slip past the score gate because the model scores high on that particular sound.

• Off by default (wake_rms_min: 0.0). Leave the entry out and the profile behaves exactly as before. Deliberately a separate parameter, not vad_voice_rms_min (which is already in use for VAD/endpointing).
• The threshold belongs in the profile, not in the bundle's manifest.yaml: it depends on microphone and gain, not on the wake word.

Finding your own value (three steps, ~10 minutes):

# 1. Record guided takes — ALL styles, especially soft/distant/turned-away:
ow-venv/bin/python -m wakeword_studio record --speaker <name>
# 2. Get a threshold suggestion from those takes alone (no daily archive needed):
ow-venv/bin/python -m tools.wake_rms_replay --nur-studio
# 3. Put the suggested value in the profile, then restart the service:
#      wake_rms_min: <value>
systemctl --user restart openclaw-voice-assist.service

The --nur-studio mode proposes round(lowest take × 0.7) and warns if the difficult styles (soft, distant, turned-away, casual) are missing — without them every proposal comes out too high and costs you soft calls later. It delivers a safe lower bound, not an effectiveness figure: without a daily archive it cannot say how many false triggers the threshold blocks. Once you have one, the full run is the better source:

ow-venv/bin/python -m tools.wake_rms_replay     # with archive: recall + precision

Example, not a preset. Our installation runs at wake_rms_min: 400 (derived from 88 genuine calls + 20 ear-checked false triggers, lowest genuine call at RMS 336, on a ReSpeaker mic with ×4 gain). It ran at 300 for the first three weeks and was raised on evidence — see below. Absolute RMS values are tied to your microphone and amplification. Copy our number onto different hardware and you lose either every call (gain lower) or block nothing (gain higher). Measure your own.

Signs the threshold is wrong: calls blocked by the gate are archived as near-misses with failed_on: "min_rms" in ~/.openclaw/workspace/wake_events.log. If intended calls pile up there, the threshold is too high for the current gain. The same signal also flags a hardware/gain change — revisit the value.

Re-check after a few weeks; that same log decides it. If the gate has blocked a good number of streaks and none of them was a call you meant, while false triggers keep getting through, the threshold is too low for your room — re-run the sweep and take the next step up. Ours went 300 → 400 that way: 16 blocked streaks in 20 days, not one of them a genuine call. Two things to stay honest about when you do this:

• Name the price. Above some value it starts costing real calls. For us 350 and 400 cost the same single call, but 400 blocked twice as many false triggers, and 450 cost six — so 400 was the knee, not a preference.
• An absolute threshold cannot fit both ends of the day. Our loudest false trigger measured 1691, the genuine call we gave up measured 336. A level gate buys you the quiet false triggers and nothing more; the loud ones are a model problem, not a threshold problem.

Keep your labelled clips — the archive deletes itself, your labels don't. triggers/ is pruned after 30 days at service start, while the labels in wake_review.jsonl live on. When the audio goes, those labels silently stop counting and the sweep quietly measures a smaller set. It cost us six ear-checked false triggers before we noticed:

ow-venv/bin/python -m tools.wake_corpus sichern   # copy labelled clips out of the pruned dir
ow-venv/bin/python -m tools.wake_corpus bilanz    # what is secured, and which labels lost their audio
ow-venv/bin/python -m tools.wake_corpus messen    # score the current bundle against that corpus

Does the gate lose calls nobody repeats? (nearmiss_shadow, optional) A near-miss followed by a trigger within seconds is a lost call — the user said it again. A user who doesn't repeat never shows up that way, and the near-miss clip itself ends at the wake word: the STT gets one isolated word and guesses. With nearmiss_shadow: true the assistant quietly keeps recording for 6 s after every near-miss (*_nearmiss_folge.wav next to the clip) and writes the STT result and the actuator's gate question to nearmiss_shadow.jsonl — only while no turn is running, without touching the Speaches cooldown, and without ever executing anything. It is a provisional estimate, not a label: every line carries "status": "vorlaeufig" (provisional), and wake_triage does not read the file. Whether STT plus "does the speaker want to switch something?" recognises real calls (questions meant for the brain slip through) still has to be measured against clips you have listened to.

And calls that never even became a near-miss? (rewind, optional) Near-miss and shadow recording only see calls where the model at least brushed the threshold. A call nothing reacts to leaves no trace. The rewind buffer therefore keeps the last minutes of microphone audio and every score in RAM and saves them on two occasions: the seconds before every trigger (if you get through after failed attempts, they are in there) and the whole buffer on a manual marker — for us a Zigbee button via MQTT, pressed a few seconds after "it didn't react". Next to each WAV sits a JSON file with the score trace, states, level per second and gaps in the microphone stream. It shows why the call failed: score near 0 (model), just below the threshold (threshold), or a gap (the microphone was deaf — not a model problem). Again: provisional, for listening, not a label. Privacy: the pre-trigger save regularly stores room conversation; agree on it with the people who live there.

Cancelling a turn while it runs (barge_in, optional)

A false wake-word trigger is not the expensive part. The expensive part is what follows it: the assistant repeats what it thinks it heard, the language model starts working, and whatever that model decides to do, it does. Until now there was exactly one way out — a stop word inside the recording, checked against the transcript. Once the recording had ended, the turn ran to completion.

That gap got wider the moment the spoken acknowledgement ("Yes?") stopped being played for single-sentence commands: without that audible cue, a false trigger is often noticed only when the assistant is already answering.

With barge_in enabled, the wake word is also listened for while the assistant itself holds the floor. Saying "stop " then aborts the running turn:

• playback stops mid-sentence,
• the heartbeat phrases stop,
• the HTTP connection to the backend is closed, which cancels the agent run server-side (documented behaviour of /v1/responses: disconnecting the client cancels the run),
• nothing is mirrored to the chat, no follow-up round is started,
• and optionally a short system note goes into the same session, so the model sees the abort in its history instead of treating the next turn as a continuation.

The stop word itself is not matched on the wake word event — it is matched on the transcript afterwards. The trigger fires on the wake word, and the "stop" you said just before it sits in the pre-roll buffer. That is why both "stop " and ", stop" work, and why a barge-in without a stop word is simply treated as a new request — the classic interruption. That request goes to the backend, never to the voice actuator: a barge-in exists to cancel, not to switch.

profiles:
  yourprofile:
    barge_in:
      enabled: true
      # Listen while the assistant itself is speaking? See the warning below.
      while_speaking: false
      # Falling two-tone + brief red LED the moment the abort fires.
      beep: true
      # Level gate for the abort. Omit it and the profile's wake_rms_min applies.
      rms_min: 400
      # Short spoken confirmation after an abort. Empty string = silent.
      ack: "Okay."
      # Post a system note about the abort into the same session.
      notify_brain: true
      # Optional: a dedicated bundle for the abort. Omit it and the profile's
      # own wake words are used.
      wakewords:
        - bundle: stopp_gaston
          min_hits: 2

Measure whether your assistant recognises its own voice before setting while_speaking: true. This is not a theoretical risk, and the two numbers below are far apart:

ow-venv/bin/python -m tools.bargein_echo_test digital --wiederholungen 5
ow-venv/bin/python -m tools.bargein_echo_test akustisch --wiederholungen 3

run	self-triggers	highest score
digital — pure TTS signal, no room, no echo cancellation	5 of 40	0.97
akustisch — real speaker → echo cancellation → real mic	0 of 24	0.07

Digitally the finding is structural: a wake-word model trained on synthetic voices recognises the synthetic voice your assistant speaks with — that voice sits inside its training distribution. It hit all three gate paths and four different sentences, including a thinking phrase with no wake word in it at all. So it is not merely the confirmation repeating the transcript.

Acoustically, the same material dropped to 0.07 on our hardware: echo cancellation takes away not just level but the wake-word character of the signal. That is why the default is while_speaking: false while our own installation runs true — the good number depends entirely on having echo cancellation in the audio path (for the ReSpeaker: only with use_speaker: true, where the XVF3800 has the loudspeaker reference). Route playback through a plain ALSA speaker instead and the digital number is the one that applies.

With while_speaking: false, self-abortion is structurally impossible — it only listens while nothing is being said — and the abort still covers the thinking and waiting phase, the one that lasts seconds to minutes. Re-measure after changing voice, volume, model or audio hardware.

Make the abort perceptible. The only signal that an abort landed is otherwise the voice stopping mid-sentence, and that is not enough: in our first live test the following recording ran for 18 seconds with no further cue. Two things fix that, and they acknowledge different facts:

• beep plays a short falling two-tone the moment the abort fires, plus a brief red LED. It says "I stopped mid-sentence and I am listening now", and it arrives immediately — no waiting for speech recognition, which takes about a second.
• ack is spoken afterwards and only when a stop word was actually found in the transcript. It says "I understood that as an abort".

Deliberately a beep and not speech for the first one: the moment the user is waiting for an answer to "did that land?" is exactly the second your TTS needs. Make it clearly different from whatever sound you use for "I'm listening" — ours is a single rising tone, the abort is a falling pair.

Keep the recording after an abort short. A barge-in is said in one breath — a stop word, or a brief new request; nobody pauses to think there. With our normal dialog endpointing (2 s trailing silence, 30 s ceiling) the recording after the abort stayed open for 18.2 seconds and picked up a bystander's question, which was then answered as a new request. Command endpointing (1 s, 8 s) is the right setting for this state.

And end it when nothing follows the trigger. "Stop Gaston" and even a fast "Gaston stop" are complete before the wake word fires, so the stop word sits entirely in the pre-roll — which the VAD never sees. "End after silence that follows speech" then waits for speech that never comes: in all three aborts in our archive the recording stayed open (LED still showing "listening") until the user said "stop" again. Our rule: after a barge-in, if one trailing-silence length passes with no speech, the recording ends as a plain abort and is never sent on as a request. Don't instead count the pre-roll as speech and let STT decide: on such a short clip it missed the stop word in one of three cases ("Gastvorstellung."), which would then have gone to the brain as a new request. Measure it with tools/endpoint_replay.py --bargein.

If you build something like this: re-check the abort flag after acquiring your audio lock. Our first live abort still spoke one sentence, because that sentence had already passed the abort check and was then blocked on the playback lock held by the confirmation still being read out. When the confirmation broke off correctly, it released the lock — and the waiting sentence went ahead with a stale check. The abort had not prevented the answer, only delayed it. One sentence is enough to make the whole feature feel broken.

What an abort cannot do: a switching command handled by the voice actuator is already executed about half a second after the recording ends. No spoken "stop" beats that. Stopping the speech does not un-switch the light. Barge-in protects you from the language model, not from a lamp.

Every abort and every near-miss is logged to wake_events.log (result: "bargein" / "bargein_nearmiss") with its own audio clip, so the gate can be swept offline like the wake word itself.

Voice actuator (optional)

Switching commands like "turn on the kitchen light" normally take the same road as any other question: through the language model in the backend. That takes seconds. The voice actuator intercepts them right after speech recognition and runs them locally — measured at about 0.5 seconds instead of several.

STT text → small LLM forms ONE JSON intent → POST /intent to your home
           automation → its reply text is read out verbatim

If the sentence is not a switching command, everything continues to the backend as before. Same if the small model is unavailable — the actuator is a shortcut, never a bottleneck.

Only a direct call reaches the actuator. A sentence spoken after the wake word or in answer to the actuator's own clarifying question may switch things. A barge-in may not — it exists to cancel, not to switch. Neither may a follow-up round after a backend reply — the microphone is open without a wake word, the person is talking to the backend, and the small model readily reads a misheard sentence as a command ("block the whole calendar" came out of speech recognition as gibberish and was executed as "start the blind stop"). Those sentences go to the backend, which can still switch through the MCP path below.

One clarifying question, then the backend. If a command is not safely executable (unknown target, group not named in the sentence, …), the actuator asks once. If the answer is not safe either, the backend decides — with both sentences and the reason as a hint (locale.unklar_hinweis), instead of the user starting over with the wake word. Reason: "set the right kitchen blind to seventy percent" was rejected twice (the model picked the group of both blinds, the group rule stopped it), while the backend later got even a garbled version right. The cost is a few seconds in exactly these cases.

Optional gate question (actuator.tor_enabled). The classifier has to commit to one target even when none is meant. With the gate on, the same small model first answers only "does the speaker want to switch something? yes/no"; anything but yes goes to the backend. On our 182 test sentences (including real turns) this took wrong switches from 3 to 0, at the cost of 16 of 86 commands taking the slow path through the backend. The model's probability for "yes" is logged but does not decide — for a small LLM it is not calibrated. The prompt is German and installation-specific (actuator.tor_prompt); measure before you rely on it in your language.

Optional second classifier: Laya (actuator.laya_url, actuator.klassifikator). Laya is an encoder with a decision head (no text) that answers gate, target and action in one pass via laya-serve; a simple parser reads the value from the sentence. Its result goes through the same check (verdict()) as the small LLM's. klassifikator sets who decides (gemma, the default, or laya); the other one judges in the shadow, never switches, and is logged next to it in actuator_schatten.log — so a candidate is measured on real sentences before it is allowed to decide. If Laya decides and does not answer (service stopped, timeout laya_timeout), the LLM decides in the same turn. With laya_rueckfrage (on by default) the assistant asks back when Laya sees an intent to switch but no device — instead of handing the sentence to a backend that would have to guess. Laya is not usable without fine-tuning: it is trained on sentences generated from your own capabilities (tools/tor_trainset.py, tools/laya_aktuator_train.py, separate venv with torch; procedure and pitfalls in LAYA_TRAINING.md). A checkpoint belongs to exactly one target list; when targets change, retrain. Nothing fails when you forget — Laya just answers worse. So the laya-serve wrapper reports the checkpoint's capabilities version in /health, the assistant compares it with the live version after every refresh and reports a mismatch (Telegram, the watcher's chat), and tools/laya_nachtraining.py (systemd timer, nightly) retrains on a mismatch: two seeds, measured against the test set next to the running checkpoint, switched only if it has no more wrong actions and the smoke test after the switch reproduces the numbers. The generator's templates are German.

Keep the model's output short. The classifier answers in compact JSON, enforced by a GBNF grammar. With a JSON schema alone the model was free to indent — 52 tokens instead of 29, twice the latency, no extra information. Generating tokens is where the time goes, so on slow hardware (we run the classifier on an integrated GPU) this decides whether the shortcut is still short.

The assistant only contains the speech side. You provide the executing side yourself: two HTTP endpoints, GET /capabilities (what may be switched) and POST /intent (do it). What you build them with is up to you — Node-RED, Home Assistant, openHAB or a fifty-line script.

→ ACTUATOR_INTERFACE.md (German) describes the contract in full, including the reasoning behind every design decision, a working minimal implementation in Flask and an acceptance check via curl.

The device vocabulary is not maintained in the assistant: schema and prompt for the small model are generated at runtime from /capabilities. Adding a new device there is enough — it becomes speakable immediately, without a restart and without touching the prompt.

The same path for the brain (MCP)

The actuator only handles what it recognises as a switching command with confidence. Everything else goes to the large model — including sentences that were meant to switch something, just phrased in a way the actuator could not place. Without a path of its own, the large model then reaches for the raw home automation API: no target whitelist, no value check, no confirmation. That is exactly how the 2026-08-01 incident behind this section happened — a misheard device name was guessed at, and every shutter in the house moved.

voice_assistant/mcp_actuator.py is a stdio MCP server that gives the large model the same guarded interface, with two tools:

There is deliberately no free-text tool: the whole point is that a device name absent from haus_ziele() does not exist — rather than being guessed at.

One rule belongs in the large model's tool instructions, because it will not arrive on its own: if an entry from namen appears verbatim in the sentence, that is the target — do not reinterpret it by word meaning as a larger group. A large model reads meaning, not names. With an everyday group "all shutters" (deliberately excluding the children's rooms) next to a group "shutters in the whole house", it picks the complete set for "all shutters" — literally, "all" does mean all. The confirmation prompt of the expensive target does not catch this: it gets confirmed, and more moves than intended. Seen in this installation five times in a row. The actuator's small model does not have the problem; it matches names. And mind the examples in the instructions themselves: a phrase like "confirmation for expensive targets, e.g. all shutters" teaches exactly the wrong equation.

Requests carry their own quelle, so the executing side applies its gate just as it does for the actuator, and reserving further quelle values for other callers stays possible. Every call is appended to the same log file as the actuator turns — otherwise the large model's switching would be invisible, which is precisely why nobody noticed the incident.

The actuator is enabled per profile; without the block it stays off:

    actuator:
      enabled: true
      base_url: "http://<home-automation>:1880/voiceact"
      llm_url:  "http://localhost:8090/v1/chat/completions"
      mqtt_host: ""          # optional: instant notification on changes;
                             # empty = polling every 10 min only

The local model is expected to be a small instruction-following LLM behind an OpenAI-compatible API (tested: Gemma-4-E2B-Q4 via llama.cpp). It must support response_format: json_schema — the closed form of the intent is half the safety, the executing side checks the other half.

The token for the endpoints lives in voiceact-token.txt in the project directory (gitignored) and is sent as the X-Actuator-Token header.

The classifier prompt is German by default — and replaceable. It is the only language-dependent part of this project. Set actuator.system_prompt in your profile to run it in English (or any other language); the code itself contains no fixed target id and no device word. Target list, contrast examples and the rule for device plurals without a room are generated from /capabilities on every refresh and substituted into the placeholders {ziel_liste}, {kontrast} and {gruppen_regel}. See config.example.yaml for the full set of keys.

If you replace the prompt, measure it. tools/actuator_grammar_test.py runs a fixed suite against the real classify() path and prints the capabilities version it was measured against — a score is only ever valid for one version of your device list. Two findings from our own measurements that appear to be language-independent: fewer few-shot examples beat more, and the plural rule only takes effect when placed after the target list, never before it. Translate the test sentences along with the prompt.

Turns handled by the actuator itself are never seen by the backend — they are written as JSONL to <workspace>/actuator_turns.log instead.

Overseer (Stage 1 — read and report only)

Since the actuator bypasses the backend, nothing double-checks whether what was said matches what was switched. The overseer closes that gap conservatively: it checks each actuator turn and reports discrepancies — it does not intervene, switch, or correct anything.

The semantic check uses an LLM (OpenAI-compatible API, like the actuator itself). A short system prompt asks: does the transcript match the intent? The LLM recognises subtler patterns that a regex heuristic would miss — negations, restrictions, prepositions vs. actions („auf 10%" is a SET action, not the OPEN action). See voice_assistant/services/watcher.py.

House knowledge (optional, haus_mcp_url). An overseer that does not know the house invents it: ours "found" a room that does not exist and explained misheard words with made-up meanings. If the home-automation side offers an MCP server with a haus_weltmodell tool (people, rooms and what the family calls them, which target is where, what house-specific targets mean), the overseer fetches that world model and puts the whole of it into its prompt — fetched by code, not by the model, because a model does not notice that knowledge is missing. Before every check it asks only for the version and reloads on change. Give the overseer its own token so the other side can tell who is asking. Without haus_mcp_url the overseer behaves as before. Measure prompt changes with tools/argus_replay.py (old vs. new prompt over the logged turns; labels for past findings live in a private test set).

Additionally, a deterministic structural check runs without an LLM:

• EXEC_DIFFERS — the home automation executed a different target/action than the intent requested.
• STATUS_PROBLEM — the automation did not answer „ausgefuehrt" (rejected, unknown target, deferred). Archived but not sent to Telegram (too noisy).

On LLM errors (provider down, timeout): the overseer sends a message to the Telegram chat („Überwachung konnte nicht erfolgen weil …") instead of failing silently. Timeout is 30s with one retry on network/timeout errors.

The overseer runs as an event-driven daemon thread — check_turn() is called immediately after each actuator turn (not on a polling timer), so a mismatch is reported within ~1 second.

One-off CLI tool (reads the log, classifies, deduplicates):

ow-venv/bin/python -m tools.actuator_watch
ow-venv/bin/python -m tools.actuator_watch --seit 3   # last 3 days only
ow-venv/bin/python -m tools.actuator_watch --alles     # re-show seen ones

Background worker inside the assistant, reporting to a separate Telegram chat (not the family voice chat):

    watcher:
      enabled: true
      chat_id: "<telegram-chat-id>"   # separate group, not the voice mirror
      quiet_start: 1                   # no messages 01:00–07:00
      quiet_end: 7
      llm_url: "https://<provider>/v1/chat/completions"
      llm_model: "<model>"
      llm_api_key: "<api-key>"
      llm_timeout: 30
      haus_mcp_url: "http://<home-automation>/mcp"   # optional, world model
      haus_mcp_token: "<own-token>"                  # not the backend's token

Only LLM_MISMATCH and EXEC_DIFFERS are sent to Telegram; during quiet hours findings are collected and held. Stage 1 is deliberately conservative — the most expensive mistake the overseer could make is a hallucinated correction, physically in the house, possibly at night. Later stages (group completion, proactive follow-up on objective signals) build on this once stage 1 has proven reliable over weeks.

Kitchen timer (optional, timer:)

"Gaston, Nudeltimer eight minutes": the assistant owns the timers (services/kuechentimer.py, state in ~/.openclaw/workspace/timer.json, survives restarts). Displaying and ringing are done by sinks: any HTTP endpoint that fulfils the contract in TIMER_INTERFACE.md (a wall tablet, a kiosk monitor, an ESP display). If no sink confirms that it is ringing, the assistant's own speaker rings.

    timer:
      enabled: true
      klingeln: 3              # rings per expiry (default; per timer by voice)
      klingel_abstand_s: 5
      nachlauf_faktor: 2       # expired timers stay visible for runtime × factor,
      nachlauf_min_s: 300      #   at least this long
      nachlauf_max_s: 1800     #   at most this long
      ansage: true             # "Nudel-Timer ist abgelaufen." on the speaker
      lautsprecher_rueckfall: true
      senken:
        - name: kitchen
          url: "http://<host>:<port>/api/timer"

Deliberate differences from the usual smart-speaker timer:

• It stops ringing by itself after klingeln rings, like any kitchen timer. "Stop" only ends the ringing early, and the timer stays visible.
• It keeps counting after expiry, shown in a different colour ("Nudel −2:30"), for twice its runtime, clamped to 5–30 minutes (an egg timer does not need to stand there red for half an hour).
• A new unnamed timer replaces the old unnamed one. "Delete the timer" without a name asks back when several exist.

Two inputs: a fixed parser (services/timer_parser.py, no model) runs before the actuator, with the same restrictions (first turn only, never a follow-up or barge-in). What it cannot read unambiguously goes to the brain, which has the MCP tool kuechentimer (same server as haus_schalten). The parser is German, like the actuator prompt. Measure changes with tools/timer_parser_test.py: FALSCH must stay 0. On recorded kitchen sentences, 31 of 40 commands took the fast path, and all misses were STT mishearing "Timer".

OpenClaw Integration

Pronunciation list (optional, aussprache:)

Piper sends every word through espeak-ng's rules for the voice's language, so loanwords and names come out wrong ("Sauce" as Sau-ke, "Andrew Jackson" as Andreef Jakson); switching voices doesn't help, they all share that front end. Piper reads text in [[ … ]] as phonemes, so the assistant replaces known problem words right before synthesis (Telegram and the brain still see plain text):

1. Base list from Wiktionary (data/aussprache/de_wiktionary.tsv, ~19 000 words, CC BY-SA): the German Wiktionary gives loanwords their usual German pronunciation — in a blind test that version won 4 of 6. Only words where espeak differs noticeably are kept; homographs (incl. across case, "Seine" the river vs. "seine"), function words and words under 4 letters are left out — measured on what the assistant actually said, without these rules the two most frequent words ("der", "Die") got wrong entries. IPA is mapped onto the symbols espeak itself produces, because that's all the voice ever heard (Wiktionary's ʁ, ̯, ʔ never occur in espeak output). Build it: tools/aussprache_grundstock.py <dewiktionary dump>.
2. Cases are collected automatically: spoken words that are neither in the list nor in the Wiktionary vocabulary (names, new anglicisms).
3. Once there are enough (min_faelle, 50), a timer runs tools/aussprache_ergaenzen.py: any OpenAI-compatible LLM (llm_url, llm_model) proposes a German respelling and an IPA per word; each is synthesised and run back through STT, and only a version that brings the word back is kept — no listening needed. If today's pronunciation already comes back, no entry is made.
4. Direct correction via the brain (voice_aussprache_setzen): the user says how it should sound in German spelling ("Sohße"), it applies at once and a sample is spoken.

Generated entries stay private; tools/aussprache_veroeffentlichen.py copies the public ones into data/aussprache/de_ergaenzt.tsv (with hard filters for speaker names and device-name parts — the LLM's own "public" flag wasn't reliable). Only for Piper voices; for another language add its list and set sprache.

English passages (englisch: true). Most song titles and artists are English, and no list can hold them all. So a German answer is scanned word by word against two word lists (Debian wamerican and wngerman under /usr/share/dict, services/englisch.py): a run of words found only in the English list is English, words in both lists right next to it join in ("Wind of Change"), unknown words (names, deliberate misspellings) take English from an English neighbour unless a German word sits next to them. German function words that are also English ("den", "in") join only when enclosed. Such a passage goes through espeak's English rules and is mapped onto the German voice's sounds — it sounds like a German speaking English, which for song titles is exactly right. Entries from eigen/ergaenzt still win; the Wiktionary base list doesn't, it reads English words the German way. Measured against a blind-labelled sample of our own music library (tools/musik_sprache_test.py, details in its docstring): about 77 % right, no German title that sounded worse.

Speaches with piper-tts 1.3.0 (e.g. speaches 0.9.0-rc.3) needs a patch. There, text after a [[ … ]] block becomes a sentence of its own; when the word ends the sentence, an empty sentence is left over and the voice turns it into a loud noise (a "hiss" after the word, inaudible mid-sentence). piper-tts 1.4.1 fixed this; for older versions the fix is patches/piper_voice.py in openclaw-voice-stack, mounted over piper/voice.py in the container.

    aussprache:
      enabled: true
      sprache: de
      llm_url: http://<llm-host>:<port>   # optional; without it cases are only collected
      llm_model: <model>
      min_faelle: 50
      englisch: true                     # optional; English passages in English (German only)

Service watchdog (optional, dienstwaechter:)

The fallback chains (STT stage after stage, Gemma instead of Laya) have a downside: a dead service becomes invisible. Ours lost Qwen-STT and Laya for 15 hours — the containers were healthy inside, but their host port forwarding had died — and every turn quietly ran on the next stage. Nobody noticed.

The watchdog checks /health of every service URL the profile uses (Speaches, llama.cpp STT, classification LLM, Laya — taken from the profile, nothing hard-coded) once a minute. A service that stays down past a grace period is reported to the overseer's chat, and if a local Podman container publishes that port, it is stopped and started again (a limited number of times, with a gap between attempts); the result goes into the same message. Recovery is reported too. Quiet hours: it heals but collects the messages.

    dienstwaechter:
      enabled: true
      ruhe_units: [laya-nachtraining.service]   # planned downtime: no alarm, no healing
      # gnadenfrist: 180   intervall: 60   heilen: true   max_heilversuche: 3
      # dienste: [{name: "My service", url: "http://127.0.0.1:9000/health"}]

If you run rootless Podman: never start a container from inside a systemd unit with a plain podman start. The port forwarder (rootlessport) stays in the caller's cgroup, and when that unit ends, systemd kills it — the container keeps running, its port is dead. That is exactly what happened to us. voice_assistant/services/container.py starts containers in their own transient scope (systemd-run --user --scope); use that.

Session Key

openclaw_session determines which session voice requests land in. For voice and Telegram chat to share context, this key must match the Telegram session key.

Find it in the OpenClaw dashboard under Sessions or in:

~/.openclaw/agents/main/sessions/sessions.json

Typical format: agent:main:telegram:group:-1003XXXXXXXXX

The script sets the HTTP header x-openclaw-session-key. Without it, OpenClaw creates a separate openresponses-user: namespace and voice turns are isolated from chat history.

AGENTS.md (voice directives)

The OpenClaw workspace file ~/.openclaw/workspace/AGENTS.md shapes how the assistant behaves on voice. Frame these as goals, not prohibitions — describe what you want to achieve so the model can act sensibly in context. A minimal voice section:

## Voice (🎤)

Messages starting with 🎤 arrive via speech recognition, and your reply is **read out loud**. That's the yardstick: talk like a person would in conversation.

- Reply in the user's language, in natural spoken sentences.
- Length follows the content — usually one to four sentences, more when the topic needs it; each sentence clear and complete.
- It's spoken, so it should sound good — leave out what can't be heard (markdown, lists, numbering, emojis).
- Transcriptions have small errors; interpret generously and act once the intent is clear — when understanding, not when actuating (see below).
- You are the keeper of the voice channel: a person is waiting at the speaker, and every second of silence feels long. When a task turns out to take longer — foreseeable up front or only mid-task — give a short spoken acknowledgement right away, do the work in the background, and announce the result via `voice_speak_text`. That mandate is about answering, not about actuating: a short question back beats a guessed device.

### Answering and actuating are not the same risk

Interpreting generously comes from the information side, and there it is right: a misheard name meets resistance — the calendar, the notes and the files hold the correct names, a wrong guess visibly fails to fit and costs one sentence. When actuating, that resistance is absent entirely. Nothing checks whether the guessed device was the intended one, and the mistake then stands physically in the room. The noun you generously skim past when answering *is* the target when actuating.

This does not imply a duty to ask back — an assistant that confirms every switch is unusable. It implies a different way of looking:

- A device name you cannot place is a finding, and it belongs in your answer — not a gap to be closed by inference until some target is left over.
- Uncertainty narrows, it never widens: "I don't know which one" must never become "then all of them". Widening the blast radius does the most damage exactly when you know the least.
- Others act in the house too — people, other voice assistants, automations. And devices need time: a shutter reporting sixty-eight percent on its way to fifty is working, not failing. Repeating a command because the target value hasn't arrived yet fights both the device and the person in the room.

### Speaker awareness & safety

Each 🎤 message is prefixed with `[Sprecher: …]`. It carries one of four things: the recognised name, `unbekannt` (recognition ran and matched nobody), `Erkennung ausgefallen` (recognition did not run at all), or `Erkennung nicht eingerichtet`. Goal: impactful or hard-to-undo actions should only happen when it's clear a trusted person wants them. For anything but a recognised name, be freely helpful with harmless things (info, status, simple queries); for anything with loss or damage potential, get confirmation from a known speaker first.

**Treat an outage as less, not more, permission.** `Erkennung ausgefallen` means nothing was measured — it is weaker evidence than `unbekannt`, not a technicality to wave through.

**Continuation has a limit.** Treating a follow-up without a fixed speaker as a continuation of the person recognised a moment ago is fine for harmless things. It does not extend to impactful actions: powering machines off, deleting, reconfiguring, flashing, acting in someone's name. Those need a speaker who is established *for that input* — not one who was established earlier. That distinction is not academic; leaving it out is what allowed the 2026-09-18 incident.

**Recording who asked needs a canonical first name.** Tools that store who wanted something (e.g. a per-person play history for music) always get the same first name per person, taken from the household list in the agent's user file, never a chat username or nickname. Sources: the recognised `[Sprecher: …]`; the owner of a personal device (a phone satellite adds a note saying which phone the input came from); in a follow-up within the same conversation, the speaker from just before; in a chat, the sender mapped through the household list. Otherwise leave the field out: a wrong name skews one person's statistics for good, a missing one doesn't. If someone asks for "my" songs and the speaker is not known, ask who is meant instead of using the whole household. The tool description of the music bridge can only say *that* a first name is needed; *which* one is house knowledge, so the mapping belongs in the agent's own instructions.

If the tool refuses, that is the rule working. Ask for confirmation by a known voice or via chat instead of looking for another route around it.

This prompt directive is guidance, not enforcement. The enforcement lives in the tool that performs the action, via [`SPEAKER_STATE.md`](SPEAKER_STATE.md).

### Mood signal (acoustic)

Some 🎤 messages carry a line with `arousal` / `valence` / `dominance` values (0–1, ~0.5 neutral) measured from the voice — the *tone*, not the content. Let it inform your picture of the person and how you act, naturally, like a human picking up on someone's tone. It's rough; interpret in context, don't over-read, and don't usually name it out loud.

### Recording hint (false trigger)

Some 🎤 messages carry a **[recording hint …]** block: the wake word is missing from the transcript and the speaker is not known. That is evidence, not an order to stay silent — usually it was a false trigger (TV, radio, a conversation in the room), sometimes a real call with a misheard wake word. Decide on the content: if the text is clearly addressed to you, answer normally; otherwise reply with exactly `NO_REPLY` — the channel then stays silent.

After a reply the microphone listens for a follow-up for a few seconds. If your reply leaves nothing open (a confirmation like "the light is on", "the music is playing", no question, no offer), end it with the word `KEIN_FOLLOWUP`: the assistant removes it before speaking and posting, and no follow-up window opens. The voice instruction that accompanies every voice turn says so; without a rule like this the window stayed open even after "I like this song". If nobody speaks after the beep, the window closes after 10 s instead of running to the 30 s cap. The assistant's own beep does not count as an answer: it reaches the microphone and looked like a short word to the VAD, so the window closed after 2 s of silence before anyone could reply. If a recording is too sparse for the STT when the silence ends it, the assistant keeps listening instead.

### Voice & speaking rate

You can freely choose and switch your own voice and speaking rate (`voice_list_voices`, `voice_set_voice`, `voice_set_speed`); pass a speaker's name as `for_speaker` to remember a preferred voice per person.

The deployed AGENTS.md holds the full version (incl. voice → chat continuation handling).

With the voice actuator enabled, one more point belongs here: clean switching commands are handled by the actuator itself and never reach the brain. A switching sentence that arrives there anyway was not recognised as a command by the guarded path, or that path was unavailable — a reason for more caution, not more ambition. Switching therefore goes through that same guarded path rather than the raw home automation API: standing later in the chain gives the brain no more powerful route, only the same one.

The recording hint block is produced by code, not by the model: with anrede_hinweis: true in the profile (default off), voice_assistant/anrede.py checks, in the first turn after the wake word, whether the wake word (or a mishearing of it, derived from the bundle name) is among the first four words. If it is missing and the speaker is not known, locale.anrede_hinweis is appended to the transcript — never in follow-ups or after the acknowledgement, where the wake word is rightly absent. Measured over 169 archived recordings: none of the 11 false triggers from TV or room conversation contained the wake word, while 60 of 69 executed switch commands did. Deliberately not a filter: a false trigger reaches the brain anyway, and it judges the content better than any rule. The default hint text is German; replace locale.anrede_hinweis for other languages.

Speaker Recognition & Enrolment

Each recording runs through Speaches diarization in parallel to STT. The dominant speaker is forwarded to OpenClaw in the wrapper prefix:

🎤 [Sprecher: jochen] How is the weather?
🎤 [Sprecher: unbekannt] How is the weather?

Faster alternative: speaker verification (sprecher_verifikation: true). Instead of a full diarization per turn (segmentation, embedding, clustering — around 2 s live, and short sentences often need a second pass), one voice embedding of the whole recording is compared with the enrolled references on the CPU (~50 ms, same WeSpeaker model). A speaker counts as known only above sprecher_schwelle and with sprecher_abstand ahead of the next-best reference. Enrolment via the brain works unchanged; a changed reference file is picked up without a restart. The default threshold (0.40) was chosen against blind, ear-checked labels so that no recording was attributed to the wrong person — measure again with tools/sprecher_verifikation_test.py for your own voices before trusting it; the speaker gate depends on it. On weak hardware (a Raspberry Pi 4 needs about 0.37 s per second of audio, longer than the assistant waits for the speaker) another machine can compute the embedding: sprecher_verifikation_url: http://<host>:8001 points to voice-analysis from openclaw-voice-stack (endpoint /fingerabdruck). The audio samples themselves are sent, so the result is bit-identical to the local computation; references and the verdict stay on the assistant, and if the service fails the speaker counts as ausgefallen (failed).

Workspace layout

~/.openclaw/workspace/voice/
  last_recording.wav             current recording (overwritten per trigger)
  current_speaker.json           who spoke last — read by gating tools
  speakers/
    jochen.wav                   active reference (sent to Speaches)
  originals/
    jochen-2026-05-09T22-15.wav  timestamped backup, never overwritten

Speaker state file — turning the label into a barrier

The recognised speaker used to exist only inside the prompt. That makes it a hint to a language model, not a rule: whether an impactful action is skipped for an unrecognised speaker was decided by prose alone. On 2026-09-18 that failed in the field — diarization was down, every turn therefore read unbekannt, and the assistant powered machines off anyway, reasoning that the request had just been made by someone known.

So the assistant now writes voice/current_speaker.json after every turn, including failed ones, and a tool that is about to do something consequential reads it and decides for itself. The full contract — the four statuses, the freshness window, a minimum implementation and an acceptance check — is in SPEAKER_STATE.md. Note what it is not: with shell access as the same user this is defence against a model talking itself into an action, not a security boundary.

Enrolment HTTP server

The voice_assistant exposes a small loopback HTTP server on 127.0.0.1:18791 that lets external tools manage speaker references:

Names are normalized (lowercase, alphanumeric + -_).

OpenClaw plugin (voice tools)

The companion plugin in openclaw-plugin/ registers the tools the LLM can call during a voice turn:

Install / register:

openclaw plugins install --link ~/openclaw_voice_assist/openclaw-plugin/
openclaw gateway restart
openclaw plugins inspect voice-enrol --runtime --json   # status should be "loaded"

Important: every tool must also be listed in openclaw.plugin.json under contracts.tools — OpenClaw (≥ 2026.6) silently refuses any tool that is only registered in index.js. After a restart, verify the gateway log has no must declare contracts.tools for: … lines.

These tools call the assistant's loopback HTTP servers (enrolment :18791, speak :18792), so the voice_assistant must be running. See openclaw-plugin/README.md for details.

Limitations

• Speaches diarization needs at least 16 kHz mono audio with 2–10 s of real speech (silence does not contribute). Very short follow-up answers (≤ 2 s) often classify as "unknown".
• A failing diarization service is reported as ausgefallen, never as unbekannt. Collapsing the two is what made the 2026-09-18 incident invisible in the log: an outage looked exactly like a stranger at the microphone.
• Recordings longer than ~10 s would OOM the GPU (Wespeaker resnet34 buffer allocation), so the diarization client truncates input + references to 8 s before the request. The original full recording is still preserved in originals/ and last_recording.wav.
• The first-time enrolment uses the same recording the user spoke their request in (Variant 1). Quality scales with recording length and noise level.

Speaches Integration

STT: POST {speaches_base}/v1/audio/transcriptions — model guillaumekln/faster-whisper-medium

TTS: POST {speaches_base}/v1/audio/speech — model speaches-ai/piper-de_DE-thorsten-medium

Diarization: POST {speaches_base}/v1/audio/diarization — models Wespeaker/wespeaker-voxceleb-resnet34-LM + fedirz/segmentation_community_1 (Speaches ≥ v0.9.0-rc.3)

60-second cooldown after connection failures. On failure the local fallback activates automatically:

• STT fallback: faster-whisper (model small, runs on the Pi)
• TTS fallback: Piper (~/.local/share/piper/de_DE-thorsten-low.onnx)
• Diarization has no local fallback — falls through to "speaker: unknown"

Other STT engines (optional)

Speaches only serves Whisper-family models well. Two profile options put another engine in front of it; each falls through to the next stage on an error, an empty result means "no speech":

stt_llamacpp_url: http://127.0.0.1:8094          # llama-server with an audio model, e.g.
                                                 #   ggml-org/Qwen3-ASR-1.7B-GGUF:Q8_0
stt_llamacpp_kontext: true                       # send wake word + device names as context
stt_llamacpp_sprache: German                     # optional: fix the language (Qwen's name for it)
stt_onnx_model: istupakov/parakeet-tdt-0.6b-v3-onnx   # onnx-asr, in-process, CPU

Order: llama.cpp → onnx-asr → Speaches → local faster-whisper. With the actuator enabled, the context is the wake word plus the first name of every target from /capabilities; the model then spells device names correctly instead of guessing. If it copies the context instead of transcribing (it happens, rarely), the same recording is transcribed again without it. Without stt_llamacpp_sprache Qwen detects the language per recording, and on quiet or unclear recordings it guesses wrong — in our archive 26 of 373 recordings came out as English, Portuguese, Chinese, Polish, Dutch …, two of them as endless loops. With the language fixed almost all of them became readable German; English words stay English ("Stop. Nächstes Lied."). Neither engine reports no_speech_prob, so the Whisper hallucination filter does not apply; false triggers reach the brain with a hint instead (anrede_hinweis).

Choose by measurement, not by leaderboard. Without reference transcripts, an STT model can still be judged by its effect: run its transcripts through the actuator and compare with labelled intents (tools/stt_vergleich.py --transkripte, candidates on another GPU via tools/stt_kandidaten.py). For plain text quality you need a few dozen sentences checked by ear — blind, without model names, and not only the ones where models disagree (that selection made one model look ten times better than it was). Run the Whisper baseline through the same Speaches you use live: the same model in another faster-whisper/ctranslate2 version, or on another GPU generation, produces different wording.

Piper TTS (local fallback)

When Speaches is unreachable, TTS falls back to Piper running on the Pi, using the models from <project>/models/piper/ (see installation step 6). The pre-rendered "Ja?" wakeword acknowledgement also uses Piper.

LED Status

WLED (mode: local) and ReSpeaker LED ring (mode: respeaker) are mutually exclusive.

ReSpeaker LED Ring — 12 phases

WLED Strip (mode: local)

Measurement tools — measure parameters, don't guess them

Every parameter that noticeably changes behaviour has a tool that proves its effect against recorded real data. Parameters are changed only against their tool, and the measured number lives in the tool's docstring — not in a chat, not in someone's head. Several of these tools appear in their respective sections above (actuator, wakeword, endpointing); this lists them all, ordered by how soon you can use them.

Four lessons shaped this discipline, all learned the hard way:

• A measurement that existed only in a scratchpad was, one day later, neither reproducible nor valid. Numbers that aren't committed alongside the tool are worthless tomorrow.
• A tool once printed its conclusion as fixed text instead of computing it — asserting an effect for four days that its own numbers contradicted. A tool must calculate its verdict from the current data, not state it.
• A tool needs to know when it has measured nothing. One of them declared a run invalid below a guessed microphone level. Measured, the signal it was supposed to detect sat right at that guessed threshold — so the tool rejected valid runs and would have hidden a real finding. If a tool has a validity criterion, that criterion has to be derived from something observable (did playback happen at all? what is this room's baseline level?), not from a number that felt about right.
• Where a generative component is involved, one run is a sample of one. Our TTS renders the same sentence differently every time (three renderings of one sentence: 137294 / 130638 / 133710 bytes). A self-trigger therefore showed up in different sentences on different runs, and a single pass over eight sentences found nothing on its first try. Repetitions are the normal case there, not a refinement.

Most of these tools need a few days of operation before they yield anything, because they build on the trigger archive and wake_events.log. On day one they look broken — they aren't, they're just waiting for material.

Raw data lives under ~/.openclaw/workspace/: wake_events.log (one line per wake decision), endpoint.log (one per recording), actuator_turns.log (one per switched turn the actuator handled itself), and voice/triggers/ (the archived wake/record/near-miss WAVs).

Day one — needs only a microphone:

• wakeword_studio record — guided real recordings of the wake word in varied styles (distance, tempo, loudness, angle). Also scores each take against the model. The foundation for everything below.

  ow-venv/bin/python -m wakeword_studio record --speaker <name>
  

• wake_rms_replay --nur-studio — suggests a level-gate threshold from those takes alone, no daily archive needed (see the level-gate section above).

  ow-venv/bin/python -m tools.wake_rms_replay --nur-studio
  

• bargein_echo_test — measures whether the assistant's own voice sets off the abort detector (see the barge-in section above). digital needs no hardware and gives the lower bound; akustisch plays through the real speaker while the real microphone listens, and that is the number that decides whether while_speaking: true is safe.

  ow-venv/bin/python -m tools.bargein_echo_test digital
  ow-venv/bin/python -m tools.bargein_echo_test akustisch
  

After a few days of operation (once the archive exists):

• wake_triage — sorts archived wake/near-miss clips into REAL CALL / NOISE / UNCLEAR, from self-labels (actions) first, STT second. Lists the UNCLEAR cases for listening. Needs trigger archive + wake_events.log + STT.

  ow-venv/bin/python -m tools.wake_triage --seit 3 --auch-trigger
  

  With --auch-trigger it also breaks the triggers down by gate path (1 frame / 2 frames / 3+ frames) against those labels, and says what turning the one-frame path off would have cost in proven real calls. That table is the only honest way to judge the short-streak thresholds: offline scorers try several frame phases and always find the best streak, so they cannot see this at all. Ours showed a path that used to buy 12 real calls and, after a retraining, bought none while still letting four false triggers through.
• endpoint_replay — replays the endpointing logic over the archived recordings and shows where a different silence/ceiling setting would have cut a recording — proving via STT whether spoken material was lost. Needs trigger archive + wake_events.log + STT.

  ow-venv/bin/python -m tools.endpoint_replay --stt
  

• wake_rms_replay (full) — measures the level gate against the archive: real calls lost (the price) vs. false triggers blocked (the gain), with a threshold sweep and Fisher exact test. Needs archive + labelled clips.

  ow-venv/bin/python -m tools.wake_rms_replay
  

• wake_corpus — lifts labelled clips out of the self-pruning archive into a permanent corpus, reports erosion (labels whose audio is already gone), and scores the running bundle against that corpus — the before-figure any retraining has to beat. --split takes the training package's manifest and separates FRESH (clips the model never saw) from TRAIN (self-measurement); without it the number is a mixture of both and claims more than it shows. Needs labelled clips.

  ow-venv/bin/python -m tools.wake_corpus bilanz
  ow-venv/bin/python -m tools.wake_corpus sichern
  ow-venv/bin/python -m tools.wake_corpus messen
  ow-venv/bin/python -m tools.wake_corpus messen --split /path/paket_manifest.json
  

  The tool's docstring also carries the recipe for A/B-ing two models on the same clips (the predecessor comes out of git history) — and the warning that this tool cannot evaluate the short-streak gate paths at all: its scorer tries several frame phases and always finds the best streak, so a one-frame trigger practically never occurs offline while live the phase is fixed.
• actuator_watch — reads actuator_turns.log and spots discrepancies (intent vs. executed, status problems). Needs actuator_turns.log.

  ow-venv/bin/python -m tools.actuator_watch --seit 3
  

• gruppenbeleg_replay — replays the group-target rule (rule A) over the real switched turns. Needs actuator_turns.log.

  ow-venv/bin/python -m tools.gruppenbeleg_replay
  

• actuator_grammar_test — the test set for the classifier prompt; re-measure after every capability change. Needs the capabilities endpoint.

  ow-venv/bin/python -m tools.actuator_grammar_test
  

• actuator_tor_test — measures only the gate question against a labelled set of real sentences (JSONL, schalten: true/false/null = the speaker's intent), including a calibration table for P(yes). You build the set from your own logs; it lives under testsets/ (gitignored — it contains your household's everyday life). Format in the tool's docstring.

  ow-venv/bin/python -m tools.actuator_tor_test
  

• aktuator_vergleich — puts both classifier chains (LLM and Laya) side by side: on the labelled set (correct / missed / WRONG per chain) or on the shadow log (agreement, disagreements to judge).

  ow-venv/bin/python -m tools.aktuator_vergleich --schatten
  

With some manual work:

• review_audio — exports clips (wake + trailing recording concatenated) for listening, and reads back the sorting as hard ear labels (wake_review.jsonl). An ear judgement outranks every automatic classification. Needs trigger archive + wake_events.log.

  ow-venv/bin/python -m tools.review_audio export
  ow-venv/bin/python -m tools.review_audio import
  

• verifier_probe — cross-checks the wake-word model on both axes (recall and precision) against the archive plus studio takes. Needs archive + studio takes.

  ow-venv/bin/python -m tools.verifier_probe
  

License

MIT — see LICENSE.