🍊 EMA AI Agent - Sherry

English · 中文 · 한국어 · 日本語

A deep role-playing AI Agent built on LangChain/LangGraph and multimodal technology.

✨ Introduction

EMA AI Agent is a highly anthropomorphic AI agent system with long-term memory and complex reasoning capabilities. It's more than just a chatbot — it's a virtual companion with an independent Persona, a dynamic Skill System, and proactive behavior through scheduled tasks and background subagents.

The Agent's character, Sherry (Tachibana Sherry), is a self-proclaimed girl detective: ever-cheerful and energetic on the outside, calm and razor-sharp at the core. The entire system is designed to support immersive, persistent role-playing with memory that accumulates across sessions.

---

🚀 Key Features

1. 🧠 Layered Memory System (Context Engine)

• Short-term Session Memory (MesMemory): conversation history persisted to SQLite (WAL mode) with automatic FTS5 indexing — including a trigram tokenizer table for Chinese full-text search; persistence runs at two timings (MessagePersistenceMiddleware): tool results are flushed the moment they return, and every model-call boundary incrementally flushes the remaining new human/ai/tool messages — write-once via the persisted_message_ids watermark — so the raw store does not depend on a compression ever firing
• History Retrieval: last-N-turns, paginated history, or turn-range queries formatted as prompt context
• Session Checkpointing: thread-safe async SQLite checkpointer (langgraph-checkpoint-sqlite) persists agent state across restarts; stale checkpoints are cleaned automatically
• Conversation Summarization: an auxiliary LLM compresses long histories mid-conversation via the Summarization middleware
• Context Governance: oversized tool results and human messages are evicted off the prompt with a recoverable on-disk copy, read_file results are sliced, a no-LLM tail clip runs before any compression, and chained summaries are filtered out of the conversation payload
• Private Knowledge Graph RAG: the multimodal_rag skill indexes documents/folders into an entity–relationship graph (vendored LightRAG + RAG-Anything on snkv vector storage) and answers via multi-hop graph retrieval
• Experience Extraction: four lifecycle paths turn conversation history into durable experience: the compression-time memory review (on every compression), plan extraction when the todo list is all-complete at a compression, the pre-compression memory flush, and the post-compression todo fork. They write to MEMORY.md / USER.md, the plan knowledge base (agent/tools/todolist/knowledge/), skills/auto/, and todos.db
• ▶️ See the Context Engine README for architecture, data models, and API details
• ▶️ See the Experience README for the extraction paths and the skill Curator

2. 🛠️ Dynamic Skill System

• SKILL.md Standard: skills are Markdown files with YAML frontmatter (name, description, optional scope: all | main_only | subagent_only) — the loader auto-discovers every SKILL.md under skills/
• Built-in Skills (skills/builtin/): cron, heartbeat, clawhub (GitHub skill installer), skill_creator (generates new skills), image_to_text, speech_to_text, video_text_to_text, text_to_image, multimodal_rag, code_wiki, llm_wiki
• Skill Management Tools: the agent can list, view, and manage skills at runtime; third-party uploads (skills/plugins/) stay inactive until explicitly enabled
• SkillSpector Security Scanning (server/service/skill_scanner.py): third-party skills are scanned by NVIDIA SkillSpector (static YARA/rule analysis + optional LLM semantic analysis via the auxiliary LLM) before activation; flagged skills are blocked from installation
• Skill Curator: the context-engine curator thread maintains auto-learned skills under skills/auto/ — see the Experience README
• Tool Timeouts: real per-tool limits come from the TOOLS_TIMEOUTS registry (WEB_SEARCH_TIMEOUT=15, TERMINAL_TIMEOUT=30, PYTHON_REPL_TIMEOUT=30); TOOL_CALL_TIMEOUT_MINUTES is a stored setting that no execution path consumes
• ▶️ See the Middlewares README for the middleware pipeline (guardrails, iteration budget, HITL, normalization, summarization, multimodal processing)

3. 🤖 Multi-level Subagent System

• 7 Runtime Tools: sessions_spawn, sessions_yield, sessions_send, sessions_kill, sessions_steer, agents_list, subagents_list
• Hierarchical Roles: depth-limited nesting (default max depth 2, hard cap 2) with MAIN → ORCHESTRATOR → LEAF roles and least-privilege tool scoping
• Isolated Context: every subagent runs with a fresh, independent context — the parent transcript is never inherited; file attachments are supported
• Reliable Delivery: results return through an EventBus announce pipeline with idempotency checks and exponential-backoff retries
• Durable Registry: run records persisted to SQLite; a sweeper recovers orphaned runs and a followup checker enforces run timeouts when configured (default: none)
• Swarm Mode: batch sub-task execution with FIFO scheduling and configurable concurrency
• Verified completion: every spawn runs an auxiliary-LLM completion judge between child turns; a continue verdict injects the judge's follow-up prompt as the next turn, bounded by the configured COMPLETION_JUDGE["goal_max_turns"] budget (default 5)
• Functional Roles (opt-in): sessions_spawn(functional_role=...) specializes a worker (general / researcher / executor / reviewer / librarian); the role drives the LLM tier, the tool allow-list, and the child's system-prompt sections
• ▶️ See the Subagent System README for the full architecture

4. 🌐 Multi-Channel Access

• Robyn Backend (server/): async HTTP API + WebSocket (/sessions/ws) on 127.0.0.1:8080, serving uploaded media under /static, /images, /audio, /video
• Desktop Client (client/): Tauri 2 + Nuxt 4 (Vue 3 + TypeScript) SPA with system tray, global shortcut, offline history cache (Dexie/IndexedDB), dark/light mode, and i18n
• QQ Bot: QQ channel adapter via the plugin system (plugins/channels/qq/)
• Message Bus (bus/core.py): internal async queues decouple channels from the agent core

5. 👁️ Multimodal Interaction

• Image Understanding (ITTT): Image-to-Text vision models for analyzing user-uploaded images
• Video Understanding (VTTT): Video-Text-to-Text models for video content analysis
• Speech Recognition (STT): FunASR-based local speech-to-text
• Text-to-Image (TTI): image generation from text descriptions via the text_to_image skill
• Document Parsing: MinerU-based multimodal document ingestion for the knowledge-graph RAG pipeline

6. ⏰ Scheduled & Proactive Behavior

• Cron Service (skills/builtin/core/cron/): one-shot (at), interval (every), or cron-expression (cron, via croniter + timezone) agent tasks, persisted to a JSON job store with per-job run history and delivery to channels
• Heartbeat Service (skills/builtin/core/heartbeat/): periodic wake-up (default 30 min) that checks HEARTBEAT.md for pending tasks, lets an LLM decide skip/run, and passes results through a notification gate

---

🏗️ Tech Stack

Built on Python 3.13 (dependency management via uv), with the following core technologies:

---

📂 Project Structure

EMA_AI_agent/
├── agent/                  # Agent core logic
│   ├── core.py             # Main agent loop (LangChain create_agent → LangGraph graph)
│   ├── wrapper/            # Graph-level wrappers (repetition guard, context limit)
│   ├── checkpointer/       # Thread-safe async SQLite checkpointers
│   ├── middlewares/        # Middleware pipeline (summarization, guardrails, HITL, ...)
│   └── tools/              # Agent-accessible tools
│       ├── subagent/       # Multi-level subagent system (spawn/registry/swarm/...)
│       ├── todolist/       # Session-scoped todo planning layer
│       │   └── knowledge/  # Plan knowledge base + `knowledge` tool
│       ├── file_tools/     # File I/O tools (read, write, patch, search)
│       ├── skill_tools/    # Skill management tools (list, view, manage)
│       ├── pub_base/       # Shared tool utilities & infrastructure (BaseSQLiteRepository, path utils)
│       ├── mcp_plugin.py   # MCP tool integration
│       ├── web_search.py   # Web search tool (Tavily)
│       ├── python_repl.py  # Python code execution
│       ├── terminal.py     # Terminal command execution
│       ├── memory.py       # Memory inspection tool
│       ├── question.py     # HITL multi-choice question prompt
│       └── message_search.py # Conversation FTS5 search tool
│
├── bus/                    # Message bus (async queues)
│   └── core.py             # MessageBus — inbound/outbound queues
│
├── channels/               # Channel interface definitions
│   ├── base.py             # Abstract channel base
│   ├── manager.py          # Channel lifecycle manager
│   └── registry.py         # Channel registration
│
├── client/                 # Desktop client (Tauri 2 + Nuxt 4, pnpm)
│   ├── app/                # Nuxt 4 SPA source (Vue 3)
│   ├── src-tauri/          # Tauri 2 native shell (Rust)
│   └── README.md           # Client documentation
│
├── config/                 # Centralized configuration (paths, feature TypedDicts, schema, settings)
│   ├── __init__.py         # API host/port (127.0.0.1:8080)
│   ├── path.py             # File path configuration
│   ├── schema.py           # Configuration schema models
│   ├── sherry_settings.py  # sherry.jsonc loader
│   └── features/           # Per-object feature TypedDicts + default instances
│
├── context_engine/         # Memory engine (MesMemory)
│   ├── core.py             # History retrieval & FTS5 search APIs
│   ├── store/              # Session message store (SQLite + FTS5, WAL)
│   ├── events/             # Append-only event log + projector
│   ├── embeddings/         # Vector semantic search (indexer / search)
│   └── curator/            # Auto-skill curation
│
├── docs/                   # Subsystem design docs (per-language READMEs)
│   ├── experience/         # Experience extraction paths + skill curation
│   ├── session_memory/     # Session-memory capabilities
│   ├── summarization/      # Compression triggers & cooldown
│   ├── loop-prevention/    # Runaway-loop prevention harness
│   ├── sandbox/            # Eval sandbox & tool isolation
│   ├── token-guard/        # 128K context-window floor
│   ├── context-governance/ # Persistence, eviction, tail clip, summary filtering
│   └── long-running-tasks/ # TaskFlow orchestration
│
├── evals/                  # Evaluation framework (dispatcher + 5 suites)
│   ├── evals.py            # Suite runner: uv run python evals/evals.py [suite]
│   ├── sandbox.py          # Sandbox that redirects repo writes during a run
│   ├── graph_rag/          # RAGAS metrics for the graph-RAG pipeline
│   ├── subagent/           # Subagent spawn-pipeline benchmark
│   ├── long_running_task/  # TaskFlow DAG eval
│   ├── session_memory/     # Session-memory stack checks
│   ├── nudge_extraction/   # AI-judged plan extraction
│   └── results/            # Per-run reports (gitignored)
│
├── logs/                   # Logging system
│   ├── logger.py           # Log configuration (loguru)
│   └── output/             # Log output directory
│
├── models/                 # Model wrappers & weights
│   ├── LLMs/               # LLM configs (main_llm.py, reasoner_llm.py, auxiliary_llm/, reasoning_* providers)
│   ├── ITTT_model/         # Image-to-Text model (cloud API or local GGUF)
│   ├── VTTT_model/         # Video-Text-to-Text model (cloud API or local GGUF)
│   ├── STT_model/          # Speech-to-Text model (FunASR)
│   ├── embed_model/        # Embedding model (local bge-m3 GGUF or cloud API)
│   ├── reranker_model/     # Cross-encoder reranker (local GGUF or cloud API)
│   ├── extract_model/      # Entity extraction model (third-party weights)
│   └── providers/          # LLM provider specifications & registry
│       └── registry.py    # ProviderSpec entries for 20+ providers
│
├── plugins/                # Plugin system
│   ├── channels/           # Channel plugins (QQ bot adapter)
│   └── mcp_server/         # MCP server configuration
│
├── pub/                    # Shared utilities & data models
│   ├── func/               # Common utility functions
│   │   ├── format/         # Text formatting utilities
│   │   ├── media/          # Media processing utilities
│   │   ├── message/        # Message processing utilities
│   │   └── validator/      # Input validation utilities
│   └── types/              # Shared data models
│       ├── message.py      # MultiModalMessage, Chat, etc.
│       ├── bus.py          # Message bus data models
│       └── client.py       # Client data models
│
├── runtime/                # Runtime state & utilities
│   ├── session/            # Session-scoped registers
│   │   ├── core.py         # Singleton SessionRegister base + per-session cleanup
│   │   ├── relation_register.py # Session/socket relation registry
│   │   ├── state_register.py   # State registry
│   │   ├── state_keys.py       # Typed StateKey registry + TypedState facade
│   │   ├── count_call_register.py # Usage/statistics counters
│   │   ├── timer_call_register.py # Timer registry
│   │   └── _callback_executor.py # Async callback executor
│   └── process/            # Process-scoped services
│       ├── crash_loop_breaker.py # Boot crash-loop detection
│       └── periodic_backoff.py   # Periodic backoff state
│
├── server/                 # Robyn backend service
│   ├── __main__.py         # Server entry point (python -m server)
│   ├── DAO/                # Data access objects
│   ├── service/            # Business logic services (incl. skill_scanner.py)
│   └── trigger/            # Route & handler registration
│       ├── http/           # HTTP endpoint triggers
│       ├── ws/             # WebSocket triggers
│       ├── channels/       # Incoming channel triggers
│       └── subagent/       # Subagent result triggers
│
├── skills/                 # Skill library (SKILL.md definition files)
│   ├── loader.py           # Skill autodiscovery & registration
│   ├── skills_snapshot.py  # Builds the skill prompt snapshot
│   ├── auto/               # Auto-learned skills (maintained by curator)
│   ├── plugins/            # Third-party uploaded skills (inactive by default)
│   └── builtin/            # Built-in skills
│       ├── core/           # cron, heartbeat, clawhub, skill_creator, image_to_text,
│       │                   # speech_to_text, video_text_to_text, multimodal_rag
│       ├── text_to_image/  # Text-to-image skill
│       ├── code_wiki/      # Codebase wiki generation skill
│       └── llm_wiki/       # Markdown knowledge-base skill
│
├── src/                    # Runtime data directories
│   ├── checkpoints/        # Session checkpoints
│   ├── data/               # Data storage
│   ├── store/              # Data stores
│   ├── rag/                # RAG index output
│   └── images/ audio/ video/ # Uploaded media (served statically)
│
├── temp/                   # Temporary files
│
├── tests/                  # Mirror-structured pytest suite (tests/<source>/...) + run_tests_split.py (marker-based runner)
│
├── workspace/              # Character profile & behavior definition
│   ├── SOUL.md             # Personality contrasts, speech style
│   ├── AGENTS.md           # Tool usage priorities, safety boundaries
│   ├── USER.md             # User-specific interaction preferences
│   ├── HEARTBEAT.md        # Pending tasks for heartbeat service
│   ├── prompt_builder.py   # Profile-to-prompt builder
│   ├── file_sync.py        # Lazy workspace template sync (per language)
│   ├── template/           # Persona templates (en / zh / ja / ko)
│   └── memory/             # Long-term memory storage
│
├── .env.example            # Environment variable template
├── pyproject.toml          # Python dependencies (uv managed)
├── uv.lock                 # Lockfile for uv
├── start.sh                # Backend startup script
└── cron_jobs.json          # Cron job schedule data

---

📚 Submodule Documentation

Each major subsystem has its own detailed README:

⚡ Quick Start

1. Prerequisites

• Python 3.13+
• uv — the dependency manager. It creates and manages .venv automatically; there is no need to create a virtual environment manually.

git clone <your-repo-url>
cd EMA_AI_agent
uv sync   # creates .venv and installs the exact dependencies from uv.lock

2. Configure Environment Variables

Copy the .env example and fill in at least the main chat model and Tavily key:

cp .env.example .env

3. Model Notes (HuggingFace Auto-Download)

Models configured for local GGUF mode are downloaded automatically from Hugging Face into models/<model>/model_weight/ on first use — no manual download is required:

• Embedding: EMBEDDING_MODEL_LOCAL=true (the default) uses the local bge-m3 Q8_0 GGUF, auto-downloaded on first run.
• Reranker: the .env template defaults to a cloud API (RERANKER_MODEL_LOCAL=false, OpenAI-compatible bge-reranker-v2-m3). Set it to true to run the local GGUF reranker instead, which is then auto-downloaded (~636 MB).
• ITTT / VTTT / Auxiliary LLM: default to cloud APIs in the template; set *_MODEL_LOCAL=true to switch to local GGUF models (also auto-downloaded).

Network access to huggingface.co is required for first-run downloads (users in China may need a proxy or a mirror). Interrupted downloads are resumed on the next start; delete models/<model>/model_weight/ to force a re-download.

4. Start the Backend

start.sh activates the uv-managed .venv and launches the Robyn backend (it does not start Ollama or any frontend):

chmod +x start.sh
./start.sh          # runs the .venv interpreter: python -m server --fast --disable-openapi

Manual start (equivalent):

uv run python -m server

The backend listens on http://127.0.0.1:8080 with a WebSocket endpoint at /sessions/ws.

5. (Optional) Desktop Client

The Tauri 2 + Nuxt 4 client lives in client/ and requires Node.js 18+, pnpm, and Rust:

cd client
pnpm install
pnpm dev          # browser mode, dev server at http://localhost:3000
pnpm tauri dev    # native desktop mode

The client connects to the Python backend at http://127.0.0.1:8080 by default (configurable via VITE_API_BACK_URL in client/.env). See the client README for details.

---

🧪 Testing

Tests live under tests/, mirroring the source tree (tests/agent/..., tests/server/..., tests/context_engine/...), and run with pytest via uv (uv run pytest for a single test file or a small selection). Every test file carries a module-level pytestmark (unit / integration / module / system / regression) that selects which runner group executes it.

Recommended: the process-isolated runner

For the full suite (and for CI), use the split runner — it executes the suite in three sequential pytest processes (never parallel), selecting tests by MARKER (not by directory), aggregates their exit codes, and prints a per-group summary plus a final verdict (exit code 0 only if all groups pass):

uv run python tests/run_tests_split.py                  # hermetic suite (default, llm_e2e excluded)
uv run python tests/run_tests_split.py --with-llm-e2e   # ONLY the real-LLM e2e tests (dedicated-job mode)
uv run python tests/run_tests_split.py -- -k spawn -q   # args after `--` are forwarded to pytest

Why separate processes? tests/agent/tools/subagent/conftest.py installs stub callables into process-global sys.modules at conftest import time. In a single-process full-suite run, pytest imports every conftest and test module during collection — before any test executes — so those stubs are live for the whole process and leak across suites: lazy (call-time) imports resolve the stub, while modules that bound the real object earlier keep stale bindings. The result is confusing, order-dependent failures in suites far away from the subagent tests (e.g. skill-scope assertions seeing a stub's fixed skill list, TypeError tracebacks naming conftest lambdas). Running the groups in separate processes makes this cross-suite pollution structurally impossible. (The stubs themselves are restore-safe since c730a46; the runner is the defense-in-depth operational layer.)

Windows note: child pytest processes get PYTHONIOENCODING=utf-8 in their environment and the runner captures their output with errors="replace", so GBK console codepages can neither corrupt the output nor crash the run.

Real-LLM e2e tests (llm_e2e marker)

Three tests in tests/agent/tools/subagent/ (test_real_e2e.py, test_spawn_direct_e2e.py) call real LLM APIs. They are:

• deselected by default (-m "not llm_e2e" — set both in pyproject.toml addopts and by the runner),
• bounded by @pytest.mark.timeout budgets (pytest-timeout): 300 s per simple test, 600 s for the concurrent test,
• run explicitly, in a dedicated job: uv run python tests/run_tests_split.py --with-llm-e2e (selects -m llm_e2e) or uv run pytest -m llm_e2e.

Expected runtimes (solo, real backend): simple task ≈ 30–60 s; complex worst case ≈ 10 min; concurrent tasks ≈ 2–9 min. A run that exceeds these budgets is a real hang, not normal slowness — the per-test timeout bounds it (300 s simple / 600 s concurrent).

CI: .github/workflows/ci.yml runs uv run python tests/run_tests_split.py on every push/PR to main, executing the suite as three sequential pytest processes (never parallel): A = unit, B = integration + module + system, C = regression. The --with-llm-e2e suite stays a separate, slower job (it costs API tokens; never run it in parallel with other suites).

Note: tests/full/ is an auxiliary/experimental directory outside the standard groups above. Every file there that drives a live LLM is tagged llm_e2e, so the default addopts deselect it and the split runner never collects it (it also --ignores tests/full/). Run one explicitly with uv run --no-sync pytest -m llm_e2e tests/full/<file>. Hermetic tests do not belong there — the real-graph HITL test now lives at tests/agent/middlewares/humanInTheLoop/test_hitl_real_graph.py and runs in the standard groups.

Evals

evals/ is a homegrown, sandboxed evaluation framework that sits beside pytest. Run every registered suite, or one by name:

uv run python evals/evals.py                # all registered suites
uv run python evals/evals.py graph_rag      # a single suite by name

Every suite runs inside evals/sandbox.py, which redirects repo writes into a temp sandbox, and writes its reports under evals/results/<suite>/<run_id>/. That directory is gitignored; per-run reports are never committed.

---

📝 Character Profile Examples

The Agent's behavior is driven by the files under workspace/:

• SOUL.md: Defines personality contrasts, speech style, and behavioral logic.
• AGENTS.md: Defines tool usage priorities, safety boundaries, and ethical guidelines.
• USER.md: Stores user-specific interaction preferences and known facts.
• HEARTBEAT.md: Lists pending tasks for the heartbeat scheduled service.
• prompt_builder.py: Builds the system prompt from the profile files.
• file_sync.py: Lazily copies any missing persona files from workspace/template/<lang>/ (selected via WORKSPACE_TEMPLATE_LANG) without ever overwriting user edits.

---

🤝 Contributing

Issues and Pull Requests are welcome! To add a new skill:

1. Create a folder under skills/ (or skills/plugins/ for third-party skills).
2. Write a SKILL.md with YAML frontmatter (name, description, optional scope) describing the skill's usage and steps.
3. Restart the Agent — the loader auto-discovers every SKILL.md and exposes it to the model. (You can also ask the running Agent to use the built-in skill_creator skill to generate one.)

Third-party skills under skills/plugins/ are scanned by SkillSpector and stay inactive until explicitly enabled.

---

Contact Information: QQ 3132225629

📄 License

This project is licensed under the MIT License.

---

💡 Tip: This project is inspired by the exploration of advanced AI agents and deep role-playing.