🤖 Animetta — AI Virtual Companion / VTuber Framework

A configurable, extensible AI companion framework.
Plugin architecture · LangGraph orchestration · Hybrid memory · Live2D-driven · Multimodal interaction

简体中文  |  English

Animetta is an open-source framework for building AI virtual companions and VTubers — characters that talk, listen, remember, emote through a Live2D avatar, and act in the world (chat, livestream, Minecraft). It orchestrates ASR → LLM → TTS → emotion as a single LangGraph state machine, with swappable @ProviderRegistry providers, hybrid memory (Chroma + SQLite FTS5 + Markdown wiki), a Vue 3 / Electron desktop app, and full-chain observability.

Why Animetta? Most "AI VTuber" projects hardcode one provider pipeline. Animetta makes every layer — LLM, ASR, TTS, VAD, memory, tools — a swappable plugin via @ProviderRegistry, with full observability built in.

Contents

• Highlights
• Architecture
• Codebase Tour
• Quick Start
• Core Modules
• Project Structure
• Extending
• Documentation
• Contributing
• Tech Stack
• License

---

✨ Highlights

Animetta is not just another "ChatGPT + TTS" glue. It is an engineered AI companion framework built around three principles: configurable, observable, extensible.

• LangGraph state-graph orchestration — not a linear pipeline, but a directed graph with conditional routing, tool-calling loops, and interrupt/resume.
• Plugin provider architecture — register new vendors via the @ProviderRegistry decorator, zero core-code intrusion.
• Hybrid memory system — Chroma vector search (70%) + SQLite FTS5 keyword match (30%) + Markdown wiki knowledge base.
• Live2D emotion-driven — LLM output → emotion analysis → Live2D parameter mapping; expressions change in real time with the conversation.
• Full-chain observability — OpenTelemetry distributed tracing + Prometheus metrics + built-in Stats Dashboard.

---

🏗️ Architecture

┌──────────────────────────────────────────────────────────────────┐
│                      Frontend (Vue 3 + Vite)                      │
│               Live2D Renderer · Chat UI · Stats Dashboard         │
└─────────────────────────────┬────────────────────────────────────┘
                              │ Socket.IO / REST
┌─────────────────────────────▼────────────────────────────────────┐
│                WebSocket Server (Starlette + Socket.IO ASGI)      │
│               Session Mgmt · Desktop App · Live2D Events          │
└─────────────────────────────┬────────────────────────────────────┘
                              │
┌─────────────────────────────▼────────────────────────────────────┐
│                   LangGraph Orchestration Engine                   │
│                                                                   │
│  ┌─────────┐   ┌─────────┐   ┌──────────┐   ┌──────────────┐    │
│  │ ASR Node│ → │ Persona │ → │ LLM Node │ → │ Emotion Node │    │
│  │         │   │  Node   │   │  + RAG   │   │ → Live2D Map │    │
│  └─────────┘   └─────────┘   └────┬─────┘   └──────────────┘    │
│                                  │                                │
│                          ┌───────▼───────┐   ┌─────────────┐     │
│                          │  Tool Node   │   │ Output Node │     │
│                          │ MC/MCP/Custom│   │TTS + Memory │     │
│                          └──────────────┘   └─────────────┘     │
└───────────────────────────────────────────────────────────────────┘
                              │
         ┌────────────────────┼────────────────────┐
         ▼                    ▼                    ▼
┌───────────────┐   ┌───────────────┐   ┌───────────────┐
│   Services    │   │    Memory     │   │   Tracing     │
│ LLM/ASR/TTS   │   │Chroma+SQLite  │   │ OTel + Stats  │
│ Live2D / VAD  │   │+ Wiki + Meme  │   │+ Prometheus   │
└───────────────┘   └───────────────┘   └───────────────┘

Deeper architecture detail: docs/architecture/overview.md.

Architecture map

The codebase decomposes into 15 layers across four areas (node counts from the architecture knowledge graph):

Backend runtime

Frontend

Configuration & infrastructure

Developer surface

---

🗺️ Codebase Tour

A ten-step reading path through the actual code, from boot to deep internals:

1. Project Overview — Start here: this README plus docs/architecture/overview.md for the purpose and shape of the system.
2. Frontend Entry Point — frontend/src/main.ts mounts the Vue 3 app (Vite + Electron); frontend/src/App.vue wires the shell that hosts the Live2D renderer, chat UI, and dashboard.
3. The LangGraph Orchestration Engine — The heart of the backend. orchestrator.py builds the directed state graph with conditional routing and tool-calling loops; state.py defines the shared AgentState that flows ASR → Persona → LLM → Emotion; llm_node.py is where generation happens.
4. Prompts & Persona — prompting/sources.py assembles the persona- and guard-aware system prompt; config/__init__.py loads character definitions and the EffectiveConfig that parameterizes every node.
5. Realtime Server (Starlette + Socket.IO) — The ASGI server bridges frontend and orchestrator: websocket.py manages sessions and streams events; routes.py declares the Socket.IO/REST route handlers.
6. Swappable Provider Services — Every capability is a plugin via @ProviderRegistry. The package roots expose the LLM, TTS, and ASR provider factories (interface → implementation → factory → export): services/llm/ · services/tts/ · services/asr/.
7. Hybrid Memory — Memory v2 (ADR-005): memory/v2/context.py blends Chroma vector search, SQLite FTS5 keyword match, and a Markdown wiki knowledge base for long-term recall.
8. Product Tools — Minecraft — The Minecraft adapter is a Node.js project living inside the Python repo, exposing world actions to the orchestrator as callable tools: tools/minecraft/.
9. Live2D Avatar & Emotion — LLM emotion output is parsed and mapped to Live2D parameters in avatar/performance.py, driving expressions in real time alongside the conversation.
10. Configuration & Tool Registry — Declarative anchors: pyproject.toml pins the Python 3.13 backend; config/tools.yaml registers the product tools the orchestrator may call.

---

🚀 Quick Start

Prerequisites

• Python 3.13 (the toolchain targets 3.13 via ruff/mypy)
• Node.js 20+ and pnpm (frontend)
• (optional) NVIDIA GPU + nvidia-container-toolkit for GPU mode

1. Install dependencies

pip install -r requirements.txt
cd frontend && pnpm install

2. Configure

Edit config/animetta.yaml to choose the persona and the complete provider map for the test, smoke, and production profiles. Provider selection lives only in this manifest; environment variables supply profile, endpoints, and secrets.

cp .env.example .env
# Choose test, smoke, or production and fill only the keys it needs:
#   ANIMETTA_PROFILE="test"
#   DEEPSEEK_API_KEY="..."
#   DASHSCOPE_API_KEY="..."
#   MIMO_API_KEY="..."
#   QWEN_TTS_API_KEY="..."

3. Run

# Backend
python -m animetta.core.socketio_server

# Frontend (in a second terminal)
cd frontend && pnpm dev

The frontend dev server runs on http://localhost:3000; the backend on http://localhost:12394.

Build from source or deploy a verified image

Build the current checkout when developing or testing local changes:

# Start or reuse the host runtimes, build animetta:local, and start Animetta.
py -3.13 scripts/runtime_lifecycle.py anima-up

Deploy an image that has already passed the main quality gate without rebuilding the application locally:

py -3.13 scripts/runtime_lifecycle.py anima-deploy --image ghcr.io/loiter74/animetta:sha-<40-character-commit>

Use ghcr.io/loiter74/animetta:main for the latest successful main build. Use the full sha-... tag, or the published @sha256:... digest, when the deployment must be reproducible or when rolling back. Private GHCR packages require a prior docker login ghcr.io; see the Docker deployment guide.

Routine py -3.13 scripts/runtime_lifecycle.py anima-down leaves the host Qwen and RVC processes and their loaded models running. Use host-tts-stop or host-rvc-stop only when GPU memory must be released. Neither host runtime is built or managed as a Docker container.

Once healthy, the frontend is served by nginx on port 80 and the backend health endpoint is at http://localhost:12394/health (also proxied at http://localhost/health).

Full deployment guides: Docker · Zeabur

---

🔧 Core Modules

Supported providers:

Socket.IO event catalog and API reference live in docs/reference/.

---

📁 Project Structure

animetta/
├── src/animetta/          # Python backend (Starlette + LangGraph + Socket.IO)
│   ├── core/              # Entry point + service container
│   ├── orchestration/     # LangGraph state graph + WebSocket server
│   ├── services/          # LLM / ASR / TTS / VAD / Singing / Meme / Live2D
│   ├── memory/            # V2 atom-based memory (Chroma + SQLite FTS5)
│   ├── tools/             # Tool calling + MCP bridge + Minecraft bot
│   ├── avatar/            # Live2D emotion/expression analysis
│   ├── config/            # Pydantic configs + provider registry
│   ├── tracing/           # OpenTelemetry observability
│   ├── notifier/          # Alert channels (Discord, Feishu, Email)
│   ├── inspection/        # Health / telemetry background checks
│   └── acceptance/        # Golden soak state machine
├── frontend/              # Vue 3 + TypeScript + Vite (Electron desktop)
├── config/                # YAML config files (personas, services, tools)
├── design-system/         # Visual design spec (HTML spec sheets)
├── docs/                  # Architecture, ADRs, deployment, references
└── tests/                 # pytest (backend) + vitest (frontend)

---

🧩 Extending

Add a provider (LLM/ASR/TTS/VAD):

# 1. Create a config class
@ProviderRegistry.register("llm", "my_llm")
class MyLLMConfig(BaseLLMConfig):
    api_key: str

# 2. Register the service
@ProviderRegistry.register_service("llm", "my_llm")
class MyLLMAgent(AgentInterface):
    @classmethod
    def from_config(cls, config, **kwargs):
        return cls(api_key=config.api_key)

Add a graph node — follow the node pattern in src/animetta/orchestration/graph/.

Add a tool — use the @tool decorator in src/animetta/tools/.

Agent conventions (for Codex / ZCode): see AGENTS.md.

---

📚 Documentation

---

🤝 Contributing

See CONTRIBUTING.md for development setup, code standards, and test commands.

---

📊 Tech Stack

---

📄 License

MIT License — Copyright (c) 2026 Cowork