PersonalAI-Assistant

A local-first, realtime personal AI assistant that listens, talks, sees your screen, and controls your PC — endless hands-free conversation like ChatGPT Voice / Gemini Live, but running on your hardware. Online providers available as an option; offline is the default.

---

Table of Contents

• Overview
• Feature Matrix vs ChatGPT Voice & Gemini Live
• Architecture
• Hands-Free Mode (VAD)
• Wake Word — "Proactive Audio"
• Screen Sharing
• Memory — Hours-Long Conversations
• Reminders & Proactive Speech
• Answer Cards
• Voice Identity & Reasoning Effort
• Providers & Profiles
• Tools & Computer Control
• Safety
• Installation
• Usage
• Configuration
• Testing
• Project Structure
• Hardware Requirements
• Roadmap
• Troubleshooting
• Acknowledgements

---

Overview

PersonalAI-Assistant combines three capabilities normally spread across separate products:

1. Realtime voice conversation — always-on listening with wake-word gating, for hours
2. Screen vision — on-demand screenshots or continuous Gemini-style screen sharing
3. Computer control — it clicks, types, scrolls, drags, and launches apps on your behalf (something neither ChatGPT nor Gemini can do)

Offline-first: every core capability runs locally with zero cloud calls. If your hardware isn't up to it, flip a dropdown to route ASR/LLM/TTS through OpenAI or Groq — same UI, same tools, same memory.

Built and tested on an RTX 5080 Laptop GPU (16 GB).

---

Feature Matrix vs ChatGPT Voice & Gemini Live

---

Architecture

┌────────────────────────────── assistant core ──────────────────────────────┐
│                                                                            │
│  👂 mic ──► VAD engine ──► wake gate ──► utterance wav ─┐  (echo-guarded) │
│  ⌨️ text ────────────────────────────────────────────────┤                │
│                                                          ▼                │
│  Provider (runtime-swappable):                                           │
│    voicechat   Nemotron VoiceChat 11B GGUF (speech→speech, one model)    │
│    modular     ASR → LLM(+memory, tools) → TTS   [local or online]       │
│    hybrid      VoiceChat voice + Qwen2-VL screen vision                  │
│                                                          │                │
│  💬 memory ◄── rolling context + summary ────────────────┤                │
│  🖥 screen share ──► latest frame for look_at_screen ────┤                │
│                                                          ▼                │
│  🧠 tool loop ──► JSON tool calls → ToolExecutor → cards/events → back    │
│                                                                            │
│  🛡️ safety ──► kill-switch hotkey + autonomy gating on EVERY action       │
└────────────────────────────────────────────────────────────────────────────┘
        │                                   │
   terminal (--hands-free)          web dashboard
                      mic ∙ wake ∙ share-screen ∙ audio queue ∙ stop ∙ cards ∙
                      transcript ∙ voices/speed/effort pickers ∙ kill button

---

Hands-Free Mode (VAD)

👂 ON → you talk → silence (~700 ms) → utterance sent →
assistant replies (mic gated) → playback ends → hangover → listening again

• WebRTC VAD (robust) with adaptive energy VAD fallback
• Speech-start confirmed after ~90 ms of voice; end-of-speech after 700 ms silence
• Toggle 👂 Hands-free in the dashboard, or python -m pai.terminal --hands-free

Wake Word — "Proactive Audio"

Arms hands-free so the assistant only responds when addressed:

📣 Armed → hears "hey assistant, what's the weather" → accepts ("what's the weather")
        → follow-ups stay active 90 s → re-arms

Two-layer hybrid:

1. openWakeWord (offline, CPU) — stock audio models: hey_jarvis, alexa, hey_mycroft, hey_rhasspy… configurable subset via wake_oww_models
2. Transcript gate (universal) — utterances without a configured phrase (wake_phrases) are dropped before reaching the model; works even with online providers

Screen Sharing

Gemini-Live-style continuous sharing:

• 🖥 Share screen toggles a change-detected capture loop (~0.7 s interval, 960 px frames) that streams live into the dashboard's screenshot panel
• The latest frame is kept server-side — the look_at_screen tool lets the assistant inspect what you're seeing on demand ("what am I looking at?")

Memory — Hours-Long Conversations

• Every turn appended to sessions/<name>.jsonl (survives restarts)
• Last N turns (memory_turns: 24) sent as rolling context each request
• Older turns fold into a running summary kept at the front of the prompt
• Dashboard: 🆕 New chat (archives), 🗒 Transcript viewer, ⬇ Export

Reminders & Proactive Speech

You: "remind me at 5pm to call mom"     → set_reminder tool → card confirms
...at 5pm...                            → assistant SPEAKS UP unprompted 🔊

• Natural-time parsing: at 5pm, at 17:30, in 10 minutes/hours (past times roll to tomorrow)
• Persistent (reminders.json) · list_reminders / cancel_reminder
• Delivered as TTS to all connected dashboards, plus a ⏰ card

Answer Cards

Floating widgets top-right of the dashboard (auto-expire ~20 s):

Voice Identity & Reasoning Effort

• Voice picker: Aria, Jenny, Guy, Ana, Sonia (UK), Neerja/Prabhat (IN) — or any edge-tts voice name in config
• Speed slider: 0.5–2×, applied across edge-tts / OpenAI TTS
• Backchannels: short "Mm-hmm?" ack clip plays while the LLM thinks (backchannel: false to disable)
• ⚡ Instant / 🧠 Deep effort: deep escalates to a bigger model via deep_llm_base_url / deep_llm_model (GPT-Live delegation parity)

---

Providers & Profiles

Brains

Profiles (where services live)

Switch profiles mid-session from the dashboard dropdown. Keys come from env vars only.

Groq note: its free tier has no TTS endpoint, so replies use your browser's built-in Web Speech synthesis.

---

Tools & Computer Control

All executions stream live into the dashboard's tool panel.

---

Safety

• Kill-switch hotkey Ctrl+Alt+Q — checked before and during every input action; drags abort mid-way; dashboard button too
• Autonomy levels: confirm (view/info only) / auto_safe (input + reminders, no shell) / full
• Process hygiene — one model process ever; provider switch unloads old first; Ctrl+C sweeps orphans and frees ports; stale-port takeover on restart
• Switch rollback + capability gating — every provider is pre-flighted before a switch (model files, GPU deps, build state). Unavailable providers are rejected with setup instructions without touching your working model; if a start fails mid-switch the previous brain is automatically restored. The dashboard dims controls that don't apply to the selected brain (e.g. voice pickers on Nemotron, whose TTS voice is baked into the GGUF)

---

Installation

cd D:\Agents-and-other-repos\PersonalAI-Assistant
python -m venv venv
venv\Scripts\activate
pip install -r requirements.txt
pip install -e .

Requirements: Windows 10/11, Python 3.10+, FFmpeg on PATH. GPU models live in D:\hoidhxd-NVIDIA-NemotronLabs-VoiceChat-11B-GGUF\ (paths in src/pai/config.py).

For local modular profile also run: Whisper server (:9000), OpenAI-compat LLM (:8081), OmniVoice (:8889). For online profiles set OPENAI_API_KEY / GROQ_API_KEY.

First run of wake word downloads ONNX models (~10 MB) automatically.

---

Usage

python -m pai.server                    # dashboard → http://127.0.0.1:8765
python -m pai.terminal                  # push-to-talk
python -m pai.terminal --hands-free     # always-listening
python -m pai.tray                      # system-tray app (server + icon)

Dashboard header: provider · profile · voice · 🔊 TTS engine · speed · effort · status · latency · hands-free badge · kill badge. Composer: text input · 🎙 hold-to-talk · 👂 hands-free · 📣 wake word · 🖥 share screen · ⏹ stop · send. Sidebar: 📸 screenshot · 🛑 kill-switch · 🆕 new chat · 🗒 transcript · ⬇ export · live tool panel.

---

Configuration

config.yaml — see the fully-commented file in the repo root. Key sections:

provider: voicechat            # voicechat | modular | hybrid |
                               # qwen_omni | glm_voice | moshi
profile: local                 # local | online-openai | online-groq

hands_free: true
wake_phrases: ["hey assistant", "assistant", "computer"]
wake_oww_models: ["hey_jarvis", "alexa"]
wake_custom_model: ""          # your trained .onnx (optional)

memory_turns: 24
session_name: default

tts_voice: en-US-AriaNeural
tts_speed: 1.0
tts_engine: edge               # edge | omnivoice | zipvoice | vibevoice | openai
tts_clone_ref: ""              # your voice wav → OmniVoice clones it
vibevoice_asr_url: ""          # optional realtime CPU ASR endpoint
voicechat_tts_mode: native     # native | edge (re-voice Nemotron replies)
backchannel: true

reasoning_effort: instant      # instant | deep
deep_model_hf: unsloth/Qwen3-30B-A3B-GGUF:UD-Q4_K_XL
deep_gpu_layers: 28            # attention on GPU, MoE experts in RAM
deep_idle_unload_s: 300        # auto-unload → VRAM returns to voice model

autonomy: full                 # confirm | auto_safe | full
kill_switch_keys: ctrl+alt+q
dashboard_port: 8765

---

Testing

python test_smoke.py         # 35 checks
python test_rollback.py      # failed provider switch restores previous brain
python test_integration.py   # dashboard HTTP + WS round-trip

Live-verified end-to-end: model boot/fallback, single-process guarantee, graceful shutdown, stale takeover, hands-free + wake gating, profile/voice/ effort switches, screen-share streaming, reminder scheduling + proactive fire, weather/clock cards.

---

Project Structure

src/pai/
  config.py              PROFILES (local/online), all runtime settings
  server.py              dashboard HTTP+WS, shared provider lifecycle,
                         hands-free + wake + share wiring, reminders,
                         graceful shutdown
  vad.py                 WebRTC/energy VAD, echo guard, mic listener
  wakeword.py            openWakeWord + transcript phrase gate (+ custom models)
  screenshare.py         change-detected continuous frame streamer
  webcam.py              OpenCV camera capture (camera-share mode)
  memory.py              ConversationStore: JSONL, rolling window, LLM summaries
  reminders.py           persistent store, natural-time parser, scheduler
  gcal.py                optional Google Calendar mirror for reminders
  stream_sanitizer.py    dual-channel FSM: strips <think>-style reasoning
                         from SSE streams (UI/TTS/tool-parser protection)
  provider_voicechat.py  Nemotron speech-to-speech (file protocol)
  provider_modular.py    ASR→LLM(+memory/tools)→TTS, local + online
  provider_hybrid.py     VoiceChat + VLM screen description
  provider_qwen_omni.py  Qwen2.5-Omni-7B GPTQ-Int4 thinker-talker S2S
  provider_glm_voice.py  GLM-4-Voice 9B int4 (emotion-steerable)
  provider_moshi.py      Moshi/MoshiVis full-duplex bridge (rust q8)
  deep_brain.py          on-demand Qwen3-30B-A3B MoE server (lazy, auto-unload)
  tts_engines.py         local neural TTS: OmniVoice / ZipVoice / VibeVoice
  tools.py               tool protocol, autonomy matrix, cards, events
  screen_capture.py      MSS→PIL fallback, cache, downscale
  input_control.py       pynput control + global KillSwitch
  audio.py               mic record / playback
static/index.html        dashboard UI
sessions/                conversation JSONL files
reminders.json           persisted reminders

Choosing a voice brain

All fit in 16 GB VRAM one-at-a-time; the server unloads the previous model before loading a new one.

---

Hardware Requirements

No GPU? Use modular + an online profile, or a small CPU LLM. Everything else (VAD, wake word, capture, tools, dashboard, reminders) is CPU-native.

---

Roadmap

• Streaming/staged TTS — SSE stream + sentence-chunked playback
• Custom wake-word models — wake_custom_model loads your trained .onnx
• Google Calendar sync — google_calendar: true mirrors reminders (gcal CLI or service account)
• Wire screenshots into VoiceChat image turns natively — shared frame auto-attaches
• LLM-backed conversation summarizer hook-up — wired with extractive fallback
• System-tray app packaging — python -m pai.tray
• Webcam input alongside screen share — camera mode via OpenCV
• Reasoning-tag stream sanitation — dual-channel FSM (stream_sanitizer.py); leakage can't reach UI/TTS/tool parser; reasoning shown in a collapsible panel
• Affective tone (emotion-adaptive prosody) — awaiting open S2S models
• Multi-user speaker identification

Architecture notes: hardening against local-runtime failure modes

Three failure modes documented in local-agent deployments, and how this project addresses them:

1. Reasoning-tag leakage (Qwen3/DeepSeek/GLM emit <think>…</think>; tags split across SSE chunks defeat string filters) → solved by the dual-channel FSM in stream_sanitizer.py: tag-open detection holds a sliding buffer, routes reasoning to a separate UI channel, guarantees clean text to the tool-call parser and TTS queue. For logit-level certainty, deep_brain.chat(grammar=...) accepts GBNF grammars that make malformed tool-call JSON structurally impossible.
2. Context-cliff / memory saturation → rolling window (memory_turns) + LLM summarization keeps prompts small; the Deep Brain runs llama.cpp with FlashAttention and q8_0 KV-cache quantization (linear KV scaling), and auto-unloads after idle so VRAM returns to the voice model. For multi-session persistence at scale, point deep_llm_base_url/llm_base_url at any server and pair it with an external memory layer (Mem0-style fact extraction is compatible with our JSONL session files).
3. Quantized-model decoding instability (broken JSON in tool calls) → mitigated two ways: sanitizer cleans the text channel before parsing, and GBNF grammar constraints are available for llama.cpp-served brains.

---

Troubleshooting

---

Acknowledgements

Nemotron VoiceChat 11B (NVIDIA) · llama.cpp · Whisper · OmniVoice · Edge TTS · Qwen2-VL · openWakeWord · webrtcvad · pynput · mss · wttr.in

---

License

MIT — see LICENSE.