Aiko-chan アイコちゃん

(Full name: Aino Aiko アイノ・アイコ aka 愛のAI子 aka 相野愛子)

A local-first AI companion with a browser WebUI + VRM avatar, optional simple CLI, persistent memory, web search, microphone input, and MioTTS voice output. Optimised for constrained hardware — runs on a Jetson Orin Nano with 8GB unified RAM.

Author: OppaAI · Beautiful British Columbia, Canada

---

Status

Phase 2 voice is implemented, and Phase 2.5 agentic workflows are now active. The default launch path is the browser WebUI/VRM frontend, including a WebSocket bridge for chat, vitals, voice status, expression, viseme, and browser microphone events. --cli remains available for simple local testing.

ASR and TTS run through the local machine by default. WebUI microphone streaming exists in the frontend/backend bridge, but full remote voice-device polish is still experimental.

Known Issues:

• TTS via MioTTS sometimes cannot inference proper voice output due to memory constraint (MioTTS and embedding models are still tuned for Jetson memory pressure; Harrier replaced BGE for better semantic separation at 640d)
• Time latency between ASR voice input ends to beginning of TTS voice output are still over 5 sec for normal chats. Need to figure out how to do proper synchronized text and speech streaming to save a couple seconds.
• ASR may have transcribing errors that output wrong text or even wrong language, especially when accent is present in speaker's voice or when using low quality microphone.
• Barge-in haven't been fully tested and may cause some runtime issues that needed to conduct more testing and debugging.

---

Demo

Click the following image to watch on YouTube ▶

---

Purpose

This project currently serves as:

• a local AI companion chatbot with persistent memory, web search, TTS, ASR, a terminal UI, and an optional browser WebUI/VRM avatar;
• a stress test for running a full conversational stack on constrained hardware such as an 8 GB VRAM GPU or Jetson Orin Nano;
• a precursor and testing sandbox for the larger Grace / AuRoRA project;
• an experimental playground for memory decay, nightly consolidation, daily reflection publishing, agentic tools, scheduled reminders, and workflow skills.

---

Features

💬 Conversational Core

• Local-first LLM — OpenAI-compatible endpoint (llama.cpp llama-server recommended), runs entirely on your hardware
• Curses TUI — Full-screen cyberpunk terminal interface with streaming tokens, status panels, and command palette
• Browser WebUI (optional) — Modern web interface with VRM avatar, WebSocket bridge, browser microphone streaming
• Persona system — Rich personality defined in persona/SOUL.md with identity, skills, and user profile

🧠 Persistent Memory

• sqlite-vec + custom Harrier ONNX embedder — Serverless vector store, no Qdrant/mem0 required
• Hybrid retrieval — KNN vector + FTS5 lexical + Reciprocal Rank Fusion
• Ebbinghaus-style decay — Memories fade naturally unless pinned
• Pinned memories — /remember command makes memories decay-proof
• Nightly dream consolidation — Midnight dream() merges near-duplicates, prunes decayed memories, pins daily summaries
• Monthly consolidation — Older full months summarized into durable pinned memories
• Encrypted storage option — SQLCipher via AIKO_SQLITE_ENCRYPTION=1 for per-user encrypted databases

🔍 Web Search & Research

• Local SearXNG — Private web search instance via Docker
• /web <query> command — Grounded answers with source citations
• Deep research tool — Multi-step search, fetch, and synthesis for agentic tasks

🎤 Voice

• ASR — SenseVoice via sherpa-onnx (int8 ONNX, multilingual JP/EN)
• VAD — Dual energy VAD and Silero VAD for voice activity detection
• Microphone — PulseAudio parec capture (local) + browser WebAudio Worklet (WebUI)
• Barge-in — Speak over Aiko mid-response to interrupt
• Speaker verification — Optional ERes2Net via sherpa-onnx speaker embeddings for owner-only voice activation
• TTS — MioTTS via llama.cpp + synthesizer server (0.4B Q4KM), bilingual JP/EN, karaoke-style text sanitization
• Staged warmup — ASR, VAD, TTS, and microphone warmed during boot

🤖 Agentic Skills

• ReAct task loop — LLM plans, calls tools, observes, repeats until done
• Toolkit modules — Web search, fetch, planning, scheduling, workspace notes, photo ingestion, repo inspection
• Skill registry — Markdown workflow definitions in agentic/skillsets/*.md
• Skill context injection — Relevant skill instructions automatically retrieved in agentic mode
• Dual-path routing — Fast semantic exemplar routing (default) + optional LLM router fallback
• Final-answer verification — Self-critique and repair loop for tool outputs
• Scheduling & reminders — Per-user schedule.json with cron-like recurrence, due announcements
• Workspace tools — Safe note-taking, file reads, photo scanning under WORKSPACE_ROOT

🌙 Nightly Dream Pipeline

• Midnight scheduler — Runs automatically at configurable hour
• Salient memory boost — Important memories reinforced
• Near-duplicate merging — Vector similarity deduplication
• Decay pruning — Low-score memories removed
• Daily reflection — Optional Hugo + GitHub Pages blog publishing
• Cross-session coherence — Ongoing improvements to memory continuity

👥 Multi-User Support (Experimental)

• OAuth identity — Provider-scoped user IDs (github_123, patreon_456)
• Per-user isolation — ~/.aiko/<user_id>/{memory.db, schedule.json, workspace/, user}
• SQLCipher encryption — Per-user encrypted databases with server-secret derived keys
• Workspace isolation — Per-user workspaces, future Google Drive mount support

🎭 WebUI / VRM Frontend (Experimental)

• three.js + @pixiv/three-vrm — Browser-rendered VRM avatar
• WebSocket bridge — Real-time chat, vitals, voice state, expressions, visemes
• Browser microphone — AudioWorklet PCM capture → dual energy VAD + Silero VAD → backend ASR
• Expression system — Idle, happy, annoyed, flustered, thinking (planned)
• Lip-sync — Viseme-driven from TTS audio (planned)

---

Documentation

---

Architecture

flowchart TD
    YOU[You] --> UI[UI\nWebUI or CLI]
    UI --> THINK[Think\nllama.cpp/OpenAI-compatible LLM]
    THINK <-->|async| MEM[Memory\nsqlite-vec + Harrier llama.cpp server]
    THINK <-->|on demand| SEARCH[Web search\nSearXNG]
    THINK --> SPEAK[Speak\nMioTTS]
    LISTEN["Listen\nSenseVoice (sherpa-onnx) + dual energy VAD + Silero VAD"] --> THINK
    THINK -.->|nightly| DREAM[Dream\nconsolidation]
    DREAM -.->|optional| REFLECT[Reflect\nHugo + GitHub]

---

Stack

---

Quickstart

Prerequisites: Python 3.12, uv, CUDA 12.6, Docker + Compose, a llama.cpp/OpenAI-compatible local LLM server, and a pulled/served chat model (3B+ recommended).

Laya conscience server (ggmlc)

The conscience layer's Laya judge does not run on stock llama.cpp — it needs the laya serve binary built from ggmlc (compiles the PyTorch Laya model to GGUF and serves /v1/decide):

git clone https://github.com/monatis/ggmlc.git
cd ggmlc
# Follow the repo's Installation & Building docs (Python package + CMake
# native runtime, CUDA backend for Jetson), then build the laya serve binary.

Deployed layout (Jetson Orin): binary at ~/laya-bin/laya-sm86, model at models/laya/conscience-phase3-ft3-q8_0.gguf, serving /v1/decide.

Full installation walkthrough → docs/INSTALL.md

git clone https://github.com/OppaAI/Aiko-chan.git
cd Aiko-chan
cp .env.example .env        # fill secrets only; edit config/*.yaml for settings
docker compose up -d
uv sync
uv run python main.py            # browser WebUI, full voice if services are available

uv run python main.py --text      # WebUI keyboard input, ASR/TTS toggled off but loaded
uv run python main.py --cli       # simple authenticated local CLI
uv run python main.py --debug     # show memory hits each turn
uv run python main.py --clear-mem # wipe all memories and exit

---

In-App Commands

---

Project Structure

Aiko-chan/
├── main.py                 # entry point; WebUI default, --cli for simple local testing
├── config/                 # category YAML settings; secrets stay in .env
├── cognition/
│   ├── think.py            # chat facade, routing, history, scheduled-job callbacks
│   └── reason.py           # shared embedding/ranking helpers
├── memory/
│   ├── memorize.py         # sqlite-vec memory, recall, pinned memories
│   ├── vecstore.py         # SQLite/sqlite-vec helpers
│   ├── forget.py           # decay scoring and cleanup gates
│   ├── consolidate.py      # periodic memory consolidation
│   ├── journal.py          # encrypted journal blob store
│   └── reflect.py          # Hugo/GitHub reflection publisher
├── sensory/
│   ├── speak.py            # MioTTS HTTP client
│   └── listen.py           # SenseVoice (sherpa-onnx) + Silero VAD
├── agentic/
│   ├── agentic.py          # ReAct task loop, tool schemas, tool dispatch
│   ├── schema.py           # graph-first playbook DAG executor
│   ├── capability.py       # capability matching for task-mode tool filtering
│   ├── experience.py       # procedural task-run experience store
│   ├── skills.py           # skill registry and workflow retrieval
│   ├── tools.py            # stable facade for pure callable tools
│   ├── wiki.py             # wiki-card retrieval for task mode
│   ├── skillsets/          # human-readable workflow documents
│   └── toolkit/            # focused tool implementations: research, planning, schedule, photo, repo, jobs
├── system/                 # config, wakeup, schedule runner, logging, userspace
├── interface/
│   ├── webui/              # browser WebUI backend + static frontend/VRM bridge
│   └── cli/                # auth and simple CLI helpers
├── persona/                # soul/personality and prompt policy files
├── wiki/                   # trusted local knowledge cards
├── docs/                   # install, architecture, roadmap, tests, history
├── assets/                 # images and VRM assets
├── docker-compose.yml
├── pyproject.toml
├── uv.lock
├── .env.example
└── README.md

---

Roadmap

Full details → docs/ROADMAP.md

---

Configuration

All non-secret runtime settings live in config/*.yaml. Environment variables in .env override YAML at runtime.

---

Architecture Changes from Previous Phases

Key architectural decisions:

• No external memory server — sqlite-vec runs in-process, zero Docker deps for memory
• Harrier over BGE — Newer 640-dim decoder-only embedder with query instructions, better semantic separation
• Custom ONNX embedder — Harrier needs last-token pooling; fastembed only exposed MEAN/CLS
• Tools ≠ Skills — Tools are executable Python functions; skills are human-readable Markdown workflows
• Single agentic loop — Semantic routing + optional LLM routing + ReAct + skills = one coherent task layer

---

Notes

• Memory uses a custom sqlite-vec backend — no Qdrant server or mem0 required. Qdrant + mem0 were dropped in Phase 2 due to OOM issues on the Jetson Orin Nano.
• Entry point is main.py, not cli.py anymore.
• LLM runtime is now an OpenAI-compatible endpoint (LLM_BASE_URL/LLM_MODEL), usually llama.cpp llama-server; older Ollama-specific settings are archived/outdated.
• TTS runtime is MioTTS server with 0.4B Q4KM model (Tried XTTS with CoquiTTS, Kokoro and RealtimeTTS, PocketTTS but removed due to Jetson OOM/latency/quality tradeoffs).
• ASR runtime is SenseVoice via sherpa-onnx + dual energy VAD + Silero VAD. (Tried ReazonSpeech K2 and faster-whisper but removed due to English capability and RAM usage tradeoffs respectively)
• Reflection publishing fails safely if GITHUB_TOKEN or GITHUB_REPO are missing.
• Multi-user isolation uses per-user directories under ~/.aiko/<user_id>/ with optional SQLCipher encryption.
• WebUI VRM frontend is experimental; remote voice device polish is ongoing.

---

Support

If you find this project useful, consider buying me a coffee ☕

Aiko social lanes (updated)

• Two-way messenger adapters are native background services for WebUI/CLI sessions via AIKO_MESSENGER_ADAPTERS; only Telegram, Discord, Matrix, and Slack are supported for conversational ingress/egress.
• Lane A1 syndicates the weekly Patreon Aiko development post on Saturdays at 18:00: full reposts go to the Hugo-backed GitHub Pages dev blog; short teasers go to X, Bluesky, Mastodon, Discord #aiko_dev, and Threads when image hosting is available. YouTube remains video-upload only, not YouTube Community teaser posting. Lane A2 manual posting remains out of Aiko.
• Lane B is the curated photo pipeline for Pixelfed posting through the social MCP server.
• Lane C YouTube video posting continues to use the existing MCP posting pipeline.
• Lane D creates one Threads teaser-list draft per night at 23:00 for tech jobs available today, using only the configured CivicJobs.ca Lower Mainland RSS and Job Bank Canada filtered RSS feeds, filtered by TECH_JOB_KEYWORDS.
• Bluesky, Mastodon, and YouTube are one-way posting tools only, not two-way adapters.
• Nightly reflection remains native GitHub/Hugo publishing rather than MCP.

Third-Party Data

This project uses the JLPT vocabulary and kanji dataset from OpenJLPT (© evanclan), licensed under CC BY-SA 4.0: https://creativecommons.org/licenses/by-sa/4.0/

Used unmodified, as distributed.

OpenJLPT itself is built from:

• JLPT level lists by Jonathan Waller (https://www.tanos.co.uk/jlpt/) — CC BY
• KANJIDIC2 (Electronic Dictionary Research and Development Group, https://www.edrdg.org/wiki/KANJIDIC_Project.html) — CC BY-SA 4.0
• Example sentences from Tatoeba (https://tatoeba.org) — CC-licensed

See OpenJLPT's own NOTICE.md for the complete field-by-field source list.

This work (the dataset as included here) remains licensed under CC BY-SA 4.0.