Project XiaoChun

A 100% browser-native anime companion — local LLM + on-device STT + EMAGE motion + Edge-TTS, zero backend

English • 简体中文

🌐 Live Demo

---

[!IMPORTANT] Notice: This repository is a browser-native AI companion research prototype. It explores 100% client-side AI stacks (WebLLM / WebGPU / ONNX Runtime Web) and Cloudflare Workers edge TTS. It is not recommended for direct commercial production deployment.

---

📖 Overview

Project XiaoChun (小蠢) is a fully browser-native anime companion. The character renders through @pixiv/three-vrm with MToon NPR shading inside an immersive linework outdoor scene; all AI inference runs in your browser tab — no Python backend, no server GPUs.

Entry is a 2D MAD preload into 3D — the camera pushes in from a distant 3.3× view while the overlay dissolves. You talk to her (ChatBar mic dictation via on-device SenseVoice, or type). She thinks (WebLLM MiniCPM5 2B q4f16_1 on WebGPU, falls back to Qwen2.5 0.5B; thinking optional), speaks (Edge-TTS over a Cloudflare Workers WebSocket), and moves with her (EMAGE ONNX in a Dedicated Web Worker). If the model is not ready yet, typed messages queue instead of getting dropped. The chat-bar menu can switch models from WebLLM's prebuilt list, or connect any custom OpenAI-compatible provider (Ollama / LM Studio / vLLM / LocalAI / cloud) via the in-app config dialog — credentials stay AES-GCM encrypted in IndexedDB.

The UI is fully SSR-hydrated multi-language (zh-CN / en / ja) via TanStack Start + i18next, and is mobile-first responsive (iOS HIG 44 pt / Material 48 dp touch targets).

---

🖼️ Preview

Entry: idol-debut style loading screen, then a dimension-break into the 3D stage

3D stage: MToon NPR character, sunny beach scene, head bubble, and chat bar

👗 Outfit Showcase

Built-in outfits, all swappable at runtime

Individual outfits

Dinner Dress	Techwear	Cheongsam	Bikini	Maid
Swimsuit	Bandage Wrap	Office Lady	Wedding Dress

---

📥 Desktop App Download & Installation

In addition to running directly in the browser, Project XiaoChun provides a cross-platform native desktop client (built with Tauri 2.0). It supports both Full-Stage 3D Mode and Frameless Transparent Desktop Pet Mode with minimal memory overhead and instant startup.

We offer three installation methods (ranked from easiest to most advanced):

Method 1: 🌟 One-Line Universal Script (Simplest · Highly Recommended)

Open your terminal (macOS Terminal / Linux Shell / Windows Git-Bash) and run:

curl -fsSL https://raw.githubusercontent.com/FireTable/project-xiaochun/main/scripts/install.sh | bash

🤖 Autonomous AI Agent Friendly: Coding agents (such as Claude Code, OpenClaw, Cursor) can run this command unattended to install XiaoChun. See Agent Skill Specification.

---

Method 2: 🍺 Homebrew Cask (Recommended for macOS Power Users)

If you are on macOS with Homebrew installed, manage installation and future updates via Homebrew Tap:

brew tap FireTable/project-xiaochun https://github.com/FireTable/project-xiaochun
brew install --cask project-xiaochun

To upgrade in the future:

brew upgrade --cask project-xiaochun

The desktop app can also check GitHub Releases in-app (top-bar ⋯ → Check for updates). Homebrew and in-app updates are both valid.

---

Method 3: 📦 Manual Release Downloads

You can download installer packages directly from the official Releases page:

👉 Go to GitHub Releases for All Downloads

⚠️ First-Launch Security Warnings Resolution:

• macOS ("App is damaged and can't be opened" or unverified developer): Because this open-source project is not notarized with Apple's paid developer certificate, macOS Gatekeeper flags browser-downloaded files with quarantine attributes (in macOS 15+, the "Open Anyway" button was even removed from System Settings). Fix:
  1. Open the downloaded DMG and drag Project XiaoChun to Applications;
  2. Open Terminal and execute:

     xattr -cr "/Applications/Project XiaoChun.app"
     

  3. Double-click the app again to launch immediately!
• Windows (SmartScreen "Windows protected your PC"): Click "More info", then click "Run anyway" at the bottom right.
• Linux (AppImage permission): If using .AppImage, run chmod +x <filename>.AppImage, then execute normally.

---

✨ Key Features

🖥️ Native Hybrid Desktop Companion (Tauri 2.x)

• Frameless Transparent Desktop Pet (桌宠): Floats borderless and chromeless directly over the desktop environment with 100% transparent operating system compositing.
• 60Hz Real-Time Canvas Alpha Bitmask Click-Through: 16ms high-frequency thread evaluates true character alpha pixels using 1-ns bitwise shift tests. Clicks seamlessly pass through transparent pixels to desktop apps beneath, while retaining solid interaction on the character and UI.
• Persistent Window Coordinates & Freeform Dragging: Built-in tauri-plugin-window-state automatically restores window position and size. Four corner handles enable fluid resizing without canvas flickering, alongside an Option/Alt + Drag shortcut anywhere on the character.
• Cross-Process Protocol (xiaochun://): Custom OS URL Scheme enabling external scripts, AI agents, browsers, and CLI tools to wake up XiaoChun and trigger speakText with full body motion and lip-syncing.
• In-app updates: Official Tauri updater reads latest.json from GitHub Releases, prompts after the splash, and replaces the whole app (not a model-file patch). Details: docs/HYBRID_DESKTOP_APP.md.

🌐 3D Holographic Interaction Guides & Physics Dynamics

• In-World Holographic Guides: Replaces flat 2D sliders with embedded 3D controls — TurnGuide3D (horizontal yaw ring with responsive light beacon), PitchGuide3D (elevation arc), and CameraYGuide3D (floating 3D height rail).
• Localized Mouse Wind Aerodynamics: Cursor velocity induces real-time 3D wind drag impulses (windForce.ts), causing hair ribbons, clothing fabrics, and accessories to flutter dynamically.
• Pure Quaternion Chain & Biomechanical Fixes: Eradicates head rotation snap and non-uniform shearing distortions through pure affine coordinates and quaternion interpolation.

🎭 VRM Core Engine & Modular Architecture

• VRM 1.0 & VRM 0.x Dual-Standard Modular Pipeline: Powered by @pixiv/three-vrm with MToon NPR shading. The core engine (VRMEngine) is decoupled into specialized subsystems, natively accommodating both modern VRM 1.0 and classic VRM 0.x avatars:
  • Transparent Pipeline Translation: All upstream motion generators (idle, EMAGE, VRMA clips, locomotion stepping, gaze) author poses strictly in the VRM 1.0 standard coordinate space. PoseBuffer automatically handles bone axis inversions and coordinate mapping under the hood, ensuring VRM 0.0 avatars work out-of-the-box with zero joint inversions.
  • Dual-Theme Linework Outdoor Scene (LineworkWorld): Pure procedural code generating a minimalist wireframe outdoor world (sun with 12 radial rays, 19 stylized skyline buildings, ground grid, and 5 tree archetypes); natively supports linework-light and linework-dark dual themes linked to the global color mode — 100% white-on-white / black-on-white, zero external textures.
  • Studio 3-Channel Lighting (StudioLighting): Streamlined directional sunlight (dir 1.10 with 2048 PCFSoft shadow camera), hemisphere sky/ground light (hemi 0.80), and back-rim fill light (fill 0.65), delivering clean and crisp anime contours without redundant lighting overhead.
  • MToon Material Manager (VRMMaterialManager): Automatic mesh semantic categorization (skin / hair / eyes / clothing); dynamic fragment shader injection with uniform uMatSaturation for live color calibration; tuned hair saturation (default 1.50) for rich anime highlights.
  • Companion Gaze Controller (GazeController): Companion eye & head tracking, physiological yaw/pitch safety clamping, micro-saccades, and thinking head sways.
  • 3D Head Bubble Tracker (BubbleTracker): 3D world-to-screen 2D projection with a 1.5px dead-zone filter, writing directly to DOM transforms to bypass 60~120 FPS React re-renders.
• Runtime VRM Upload: Top-bar upload button accepting arbitrary standard VRM 1.0 or VRM 0.x avatars with automated full-pipeline motion and ground-anchor adaptation.

👗 Atomic Outfit Swap & Delta Addons

• Zero-Frame T-Pose Pop-in & Seamless Hot Reloading: Captures a millisecond-level snapshot of bone rotations, expression weights, gaze coordinates, and playback timestamps before swapping. The new VRM is built in the background, pose-restored in memory before scene mounting, and atomically swapped within a microtask — guaranteeing 0 frames of T-pose glitching.
• Delta Patch Distribution (.vrmbase + .vrmaddon): Eliminates bloated multi-megabyte model downloads by stripping shared body geometry into a compact base model xiaochun_base.vrmbase (5.9 MB, oxipng texture-compressed); 9 distinct wardrobe sets (Dinner Dress, Techwear, Cheongsam, Bikini, Maid, Swimsuit, Office Lady, Wedding Dress, Bandage Wrap) are distributed as incremental .vrmaddon patches (14 MB each, 13 MB for grand wedding gown).
• Web Worker WASM bspatch Assembly: Reconstructs the complete VRM in a Dedicated Web Worker using WASM bspatch with zero main-thread frame drops.
• Two-Tier IndexedDB Cache: Base models and patched VRMs are locally cached in IndexedDB (xiaochun-vrm-cache) for instant subsequent swaps. Detailed in docs/OUTFIT_SWAP.md and docs/VRM_BUILD_WORKFLOW.md.

🔮 Cinematic Post-Processing Pipeline

• UnrealBloomPass Anime Halos: Renders soft bloom glow (default strength 0.35, radius 0.65, threshold 0.75) for luminous character highlights.
• Background Luminance Bypass: Solves over-exposure in bright linework scenes by selectively applying bloom to character hair and clothing while rejecting high-luminance background linework.
• ToneMapping Capsule Grid: Integrates 5 film curve mappings (ACESFilmic, Neutral, Cineon, Linear, None) with real-time exposure tuning.
• Full-Spectrum Color Grading (BC/HS): A unified single-pass fullscreen shader controlling Brightness, Contrast, Saturation, and Hue Shift. Full technical specifications in docs/POSTFX.md.

🧍 Biomechanical Bone Morphing & Decoupling Engine

• 28-Parameter Orthogonal Morphing: Complete live anatomical customization across head, neck, shoulders, torso thickness, waist, belly, hips, buttocks, bust, arms, fingers, thighs, calves, and feet. Shoulder width defaults to 110% (1.10) with an expanded range up to 250% (2.50) to support diverse proportions from slender anime figures to tailored jacket silhouettes. UI groups the 28 sliders into 7 body regions inside sub-cards with a search filter.
• Anatomical Boundary Locking: Converts symmetrical center-out bone scaling into directional expansion (e.g., torso thickness expands posteriorly, locking the anterior chest and abdomen flat; buttocks volume expands backwards without pelvic protrusion).
• Procedural Abdominal Morphing (belly): Deforms 687 front abdominal wall vertices with smooth cosine falloff ($0.70\times \sim 1.80\times$), completely decoupled from Spine bones — zero impact on lumbar thickness or spinal curvature!
• Knee-Foot Ground Anchor Alignment: Broadens saddlebag hips while automatically neutralizing lateral femur shifts at the knee, keeping legs strictly plumb and feet planted together.
• Shear-Free Head Composition: Isolates the Head world matrix from non-uniform neck scaling, preventing eye flattening or hair slant.
• Dynamic Crown Height Measurement: Live millimeter-accurate character height dynamically derived from vertex meshes across shoes, FootIK sink, and morph slider changes. Displayed in the drawer header chip and the optional on-canvas floating ruler (visible while the drawer is open on non-mobile).
• Technical Standard: Full mathematical specifications, formulas, and parameters matrix available in docs/BONE_MORPH.md.

🩰 Universal Motion Blending Pipeline & Ground Dynamics

• Zero-Friction Any-Motion Ingestion (playMotion): Ingest VRMA URLs, raw ArrayBuffers, or THREE.AnimationClips through a single call; automatically performs humanoid retargeting, hips normalization, and supports whole-body (all) or upper-body (upperBody) masking.
• Quintic Smootherstep Inbetweening: Based on $6t^5 - 15t^4 + 10t^3$ curves over a unified 0.75s window with strictly continuous velocity and acceleration. Detailed in docs/MOTION_PIPELINE.md.
• Biomechanical FootIK & Ground Anchoring: Two-bone analytical IK during EMAGE speech; plants from the previous motion's feet (footIkIdlePlant), yields completely to BodyTurn strides, then recaptures after the step. Shoe-off sink 4.6cm/3.9cm. Detailed in docs/FOOT_IK.md.
• Procedural Locomotion Stepping & Gaze: 4-phase stepping state machine with spring yaw tracking and bio-saccades. Detailed in docs/BODY_TURN_AND_GAZE.md.
• Layered Pose Evaluation Graph:
  • Layer 0 (Base): NaturalIdleSystem procedural breathing, 8-figure pelvic postural sway, and relaxed biomechanical finger curling;
  • Layer 1 (Main Action): VRMA thinking loops, EMAGE streaming gestures, and universal clips crossfading smoothly;
  • Layer 2 (Locomotion): BodyTurnSystem procedural stepping on LEGS_MASK only (no hip rotation, no spine yaw);
  • Compose: composeLayeredSmooth after FootIK + Gaze so ground plant and look-at are follow targets, then commitToVRM.
• Architecture Rules & Lifecycle Order: Full 10-step render loop lifecycle and developer anti-patterns detailed in docs/ARCHITECTURE_AND_RULES.md.
• Loading Screen: layout, visual layers, motion/performance rules and copy for the splash are documented in docs/LOADING_OVERLAY.md.
• Technical Docs Center: Complete architecture sitemap and agent navigation available in docs/README.md.
• Tauri vs Web vs /embed differences: side-by-side table (window / background / click-through, gestures, UI, camera, persistence, loading, security, build & test) with code locations and maintenance rules in docs/PLATFORM_DIFFERENCES.md.

🧠 100% Browser-Side AI Stack

• LLM (WebLLM, default) — @mlc-ai/web-llm MiniCPM5 2B (q4f16_1) on WebGPU (fallback Qwen2.5 0.5B). Lightweight and responsive on mobile and low-VRAM devices; chat-bar menu supports live model switching and thinking mode toggles; response language adaptively mirrors the user's prompt.
• LLM (Custom OpenAI-Compatible Providers, optional) — Connect any OpenAI-compatible HTTP service via the in-app config dialog: Ollama / LM Studio / vLLM / LocalAI / cloud (OpenAI, DeepSeek, Qwen API …). Provider profiles are AES-GCM encrypted in IndexedDB; the active provider is one click away from switching. When a custom provider is active, WebLLM is not preloaded — saves 1-2 GB VRAM on local and avoids wasting bandwidth on a model you won't use.
• Unified Provider Factory (chatWorkflow.runChat) — Same-shape runChat(opts) → string contract for both WebLLM and custom providers; the dispatcher picks one per request via a ChatProvider registry. Adding a new provider = drop in a descriptor.
• User-Customizable System Prompt, Memory Turns & Head Bubble — Open the chat-bar menu → 对话设置 / Chat Settings to override the character system prompt (free-form text, falls back to default when empty/equal), tune the conversation memory-turn count (1–50, default = device-recommended), and toggle the head chat bubble (APP_CONFIG.chat.showHeadBubble, default on). When off, HeadBubble does not render; ChatDirector / bubbleTracker events still fire. Overrides persist in IndexedDB (xiaochun-user-settings); turn bounds live in APP_CONFIG.memory.userTurnsMin/Max.
• Motion — EMAGE full-body co-speech motion (ONNX Runtime Web) in a Dedicated Web Worker: wasm execution provider + INT8 (useInt8); does not use WebGPU (model tensors include int64). Temporal Gaussian smoothing and natural idle blends. Streaming windows T=64 with per-window motion_chunk for TTFA; A/V hold until audible TTS; hop/seam tunables under APP_CONFIG.emage.motion (advanceFrames 60..64). See docs/EMAGE_MODEL.md. The shipped model set is a slimmed, INT8-quantized and graph-optimized build (download 191.35 → 71.59 MB, no retraining), produced by packages/emage-onnx; see docs/PERFORMANCE.md.
• TTS — Edge-TTS 晓伊 (XiaoyiNeural, zh-CN, +10 Hz) via a hand-rolled native WebSocket client in src/lib/edge-tts-core.ts (no third-party TTS SDK); emoji stripped before speech.
• STT — SenseVoice Small int8 (zh/en/ja/ko/yue) in a Dedicated Worker: energy VAD auto-segments utterances, recognizes on-device, inserts at the ChatBar caret. CDN weights under cdn.firetable.tech/xiaochun/stt/…-2024-07-17. See docs/STT.md.
• LLM + TTS + EMAGE orchestrated by the chat director on the main thread at 60 FPS. LLM may use WebGPU (WebLLM); EMAGE stays on wasm/INT8.

⚡ Streaming Speech & Adaptive Gesture Pipeline

• Smart Speech Chunking: Eliminates long-text generation wait bottlenecks by segmenting speech into natural 30~60 character clauses split at semantic punctuation (.!?\n or natural comma pauses). xc.say has no character cap; this slicer is the limit.
• Speech handoff: A newer say / speakText / speakAudio fades the line still playing (about 0.4s, skipped when the gain is already 0) and then starts the new one. The previous pipeline exits on its utterance id, so the two do not play together. stop() still cuts immediately.
• Zero-Latency Concurrent TTS Prefetching: Downloads audio for all chunks concurrently via non-blocking network I/O, flattening TTS latency to 0ms.
• Dual-Condition Pre-buffering: Balances chunk ratio ($\lceil N / 3 \rceil$) with an upper-bound cap (max 2 chunks, 812s of audio). 1~2 segments start almost instantly; long paragraphs begin playback as soon as 2 chunks are ready while subsequent motions stream in the background.
• Continuous Latent Autoregressive Seed Carryover: The Dedicated Web Worker retains the 4-frame latent seed (continueFromPrevious) across chunks, making multi-chunk generation mathematically identical to a single long-run autoregressive inference.
• Streaming EMAGE (motion_chunk): Worker steps T=64 audio/motion windows and posts transferable motion_chunk after each successful runStep so gestures start before the full utterance finishes (TTFA). First chunk is buffered until TTS AudioContext.start (releaseMotionForAudio) so motion never leads audible audio.
• P0b isolation + wasm threads: Document responses set COOP same-origin + COEP credentialless (src/server.ts, Vite preview/dev, public/_headers) so crossOriginIsolated enables SharedArrayBuffer and ORT wasm multi-thread when the browser isolates; scratch buffers are reused across windows.
• P0c / E1+E2 hop & seams: Chunk seams and PCM hop are driven by APP_CONFIG.emage.motion — advanceFrames (60..64), chunkSeamMaxFrames, seamJumpThreshold / seamJumpFramesScale, pose micro-fade knobs.
• Physiological Angular Velocity Limiting Transition: Biomechanical angular velocity limits (arms 2.2 rad/s, neck/head 1.6 rad/s, torso 1.2 rad/s) and critical spring damping ensure natural, snap-free transitions.
• Adaptive Conversational Idle (SpeakIdleSystem): Characters adaptively respond to the current gesture during inter-segment pauses — high gestures hover with breathing buoyancy and gentle micro-settling (>1.5s); fingers flex along the anatomical Z-axis; awareness gaze drifts and micro-nods eliminate frozen mannequins.
• Live Pipeline Console Table Tracker: Real-time console.table monitors chunk TTS, EMAGE inference, playback progression, and transition modes.
• Head Bubble Progress Indicator: A pulsing progress pill (🟢 1 / 5) in the bubble status bar cleanly displays segment progress without intruding on dialogue text.

🔄 Cross-Device Sync (Pure-Front-End)

• Text-Only Transfer — Open the chat-bar menu → 跨设备同步 / Cross-Device Sync → pick what to sync (custom providers + active model / thinking mode / chat settings) → an AES-GCM-256 encrypted blob (xs:v1:iv.ct.key) is generated and put in a copyable text block. The key travels inside the ciphertext — paste once on the other device, no separate key channel needed.
• Provider Config Sync — Custom OpenAI-compatible providers (including their AES-encrypted API keys) round-trip across devices via listProvidersDecrypted() on the sender side and saveProvider() re-encryption on the receiver side, using each device's local salt.
• Preview-Before-Apply — The receiving device decrypts locally, shows exactly what will change (active service / N providers / thinking on-off / chat settings including head-bubble on-off), and only commits after the user clicks 确认导入. Import writes userSettings through saveUserSettings so the in-memory cache and HeadBubble subscribers update immediately.
• Pure Client-Side Architecture — Pure client-side execution with no third-party telemetry and no backend coordination.

💾 Client-Side Multi-Tier Memory System

• 100% Local Privacy (IndexedDB): Powered by browser-native IndexedDB (xiaochun-memory database). Dialogue turns, personal preferences, and recalled facts stay entirely on the client device — zero telemetry or chat logs sent to any server.
• 3-Tier Memory Architecture:
  • Short-Term Turns: Automatically buffers the most recent conversational rounds (default 6 turns, clipped to 180 chars per turn via APP_CONFIG.memory.shortTermTurns), maintaining dialogue coherence.
  • Entity Profile Extraction: Multilingual (zh / en / ja) regex smart extraction captures user names/nicknames (e.g., "my name is...", "call me...", "我叫...", "叫我...", "私は..."), likes, dislikes, and entity facts with deduplication.
  • Long-Term Notes & n-gram Retrieval: Archives conversational summaries (up to 80 notes); retrieves the most contextually relevant Top-K (default 4) past memories using an n-gram similarity scoring algorithm (gramScore).
• Non-blocking Async Persistence & Dynamic Injection:
  • Async Commit: Executes rememberTurn() in the background upon turn completion, never stuttering UI animations or audio playback.
  • Contextual Prompt Injection: recallForChat() swiftly retrieves entities and relevant notes, injecting localized memory headers (appendMemoryToSystem) directly into the LLM system prompt so XiaoChun naturally remembers your identity and prior topics while prioritizing current chatter.

🌐 i18n & SSR

• TanStack Start full-stack SSR with cookie-based language hydration — no client-side language flash.
• 3 languages shipped: 简体中文 / English / 日本語.
• System prompt — XiaoChun is a companion; she replies in the language of this user message, not the UI language.
• shadcn-style dropdown in the top-right for language switching (Radix UI primitive).

📱 Mobile-First UI

• iOS HIG 44 pt / Material 48 dp touch targets throughout.
• Bottom chat input ≥ 40 px with 16 px text (prevents iOS Safari auto-zoom on focus).
• Tw + tailwindcss-animate micro-interactions; liquid-glass aesthetic.
• Top-right action bar — upload VRM, language, GitHub; the debug drawer only appears on localhost.
• Chat bar — queue-while-loading, model/thinking menu, send.
• Floating speech bubble writes transform on the DOM with a deadzone, so React does not re-render at 60 FPS.
• Mobile — preload stickers stay; the chat bar clears the home-indicator inset.

🛠️ Dev Tooling

• Debug drawer (isDev(): Vite DEV or browser loopback, not Tauri production) — 8 sections in fixed order (🎭 预设表情 → 🎥 镜头设置 → 🎨 画面色彩 → ✨ 骨骼体型 → 🧩 模型部位 → 💡 灯光通道 → 🔮 画面后期 → ⚡ EMAGE Perf). Each section owns its own React state, has its own reset, and shows a "modified" dot only when the user has changed something since mount. Schema-driven render: add a section = one entry in SECTIONS + one in the component REGISTRY. PostFxSection manages bloom intensity/radius/threshold, tone mapping modes, and color grading matrix; EmagePerfSection surfaces wasm_env / isolation / numThreads / stage timings for P0b acceptance screenshots. See docs/ARCHITECTURE_AND_RULES.md §3 for the full primitive breakdown.
• ChatBar test-speak dropdown (dev) — spring / 蜀道难 presets with live segment counts via splitIntoSpeechChunks, for end-to-end TTS+EMAGE streaming checks without typing.
• Per-frame slider drag: sliders split per-tick (engine) from commit-time (state + localStorage) so a 28-slider BoneMorphSection doesn't re-render its siblings on every pointer event. Display numbers next to each slider follow the thumb in real time via imperative textContent writes from SliderWithAnchors' liveValueRef mechanism — React reconciliation is bypassed entirely.
• Camera section: FOV slider (with hover ⓘ tooltip listing 20°/30°/45°/60° reference values), 📷 min / 🔭 max camera distance sliders (mouse wheel + pinch zoom range), and the body-turn toggle. Default push-in distance is computed from FOV via defaultShotExtent so the framing stays consistent at 15°/20°/60° (no more "long-lens crops the face").
• Bone morph section: 28 sliders grouped into 7 body regions (整体 / 头颈 / 躯干 / 臀部 / 胸部 / 上肢 / 下肢), each as a sub-card; shoulder width defaults to 110% with a maximum range of 250%; search filter hides empty regions.
• Wardrobe section: 3-state per-part rendering — 穿 (toggle on) / 未穿 (toggle off, line-through) / 未装配 (dashed disabled div, no checkbox). "Equipped" comes from vrmEngine.materialManager.partMaterials[id]?.length, not the user's visibility toggle. Note: For full wardrobe changes, use the top navigation bar's outfit selector.
• Cloudflare Workers (src/server.ts) — handles full-stack TanStack Start SSR alongside native WebSocket streaming for Edge-TTS; also applies COOP/COEP credentialless on HTML/SSR document responses (Workers+Assets ignores public/_headers for documents) so EMAGE ORT wasm can use SAB threads when isolated.
• Vite dev middleware (vite/localApiPlugin.ts) — local development powered by Miniflare runtime for 100% dev/prod parity. Forwards GET /api/tts to your TTS_PROXY_URL (e.g., the deployed Worker) when set, else opens a local native WebSocket to Edge-TTS (src/lib/edge-tts-core.ts).
• Single source of truth: src/config.ts consolidates lighting / camera / expressions / saturation / LLM / R2 model config / 28 bone morph params.

---

⚡ Performance & Size

• EMAGE download 191.35 → 71.59 MB (brotli per file 161.3 → 57.7 MB), with no retraining: slim step (face branch removed, seed computed in JS), INT8 Conv weights, cross-attention layers 0–3 removed, offline graph optimization. Node wasm window 548 → 274 ms (1 thread) and 165 → 79 ms (4 threads); session create in Chrome about 196 → 66 ms (offline graph optimization). Quality is checked at logits/argmax level only (top-1 about 92.5% vs FP32); phones, Safari and Firefox are not measured.
• Parallel model download with pipelined session create (session create stays serial); models load from cdn.firetable.tech/xiaochun/emage/ into the emage-models-v2 cache, old caches are purged.
• @mlc-ai/web-llm is lazy-loaded: the shared vrmEngine chunk shrinks 6.95 → 1.01 MB and /embed no longer requests web-llm unless chat / eager preload is used.
• Opt-in multi-threaded EMAGE in iframes: SDK option crossOriginIsolated (default off; the host needs COOP same-origin + COEP credentialless/require-corp).

Numbers, methods, what was tried and rejected, and what is not measured: docs/PERFORMANCE.md (Chinese).

---

🧩 Embed in Your Website (npm)

Put XiaoChun on your own site with a few lines: a lazy, origin-checked <iframe>, a framework-free SDK, a <xiaochun-avatar> Web Component, React bindings, and speakAudio() for feeding your own audio. Pass audible: false when the host page plays the sound and XiaoChun should only move and lip-sync. For that audio, utterance start fires when the clock starts, after the thinking pose and the first motion window. playbackRate (0.25–3) and volume (0–1) scale host audio and its motion clock; setPlaybackRate() / setVolume() change them while it plays. TTS is unchanged.

npm i @firetable/project-xiaochun

Also built in (all opt-in or sensible defaults, details in the package docs):

• Move / resize the iframe with the mouse: draggable (left-drag the avatar) and resizable (drag a corner; default min 120×180, max = viewport), driven by the same gesture state machine as the desktop app (src/core/gesture); long-press / Cmd / Ctrl still do the 3D turn. Runtime setDraggable / setResizable / setSize / getBox.
• Rounded corners like the desktop window: opaque scenes get a 20 px radius by default (borderRadius option, setBorderRadius(), border-radius attribute, --xc-radius).
• The iframe remembers the outfit and scene itself (xiaochun_wearing_outfit / xiaochun_scene_theme in its own localStorage); priority is URL / SDK options > saved > default; persist optionally mirrors them in the host page.

👉 Package docs: packages/project-xiaochun/README.md · Design & protocol: docs/EMBED.md

---

🛠️ Tech Stack

---

🚀 Quick Start & Deployment

Requirements: Node.js 18+, package manager: pnpm (enforced via preinstall hook).

Local Development

# 1. Install dependencies
pnpm install

# 2. (Optional) Configure environment — copy .env.example to .env.local and edit
cp .env.example .env.local
# - VITE_EMAGE_BASE / VITE_EMAGE_BASE_PROD: ONNX model base URLs
# - TTS_PROXY_URL: Forward Edge-TTS to your Cloudflare Worker (auto-appends /api/tts)

# 3. Start dev server (SSR + Miniflare local edge runtime)
pnpm dev          # → http://localhost:5185

# 4. Production build & preview
pnpm build        # outputs dist/client & dist/server (Worker)
pnpm preview      # preview production Worker behavior locally

Connecting a Custom LLM Provider

Open the in-app Custom Model Service dialog (chat-bar menu → provider icon, or top-bar gear). Pick a template (Ollama / LM Studio / vLLM / LocalAI / Generic OpenAI), fill the base URL and (optionally) an API key, hit Test connection, then Save & enable. Active providers appear with a folded-corner checkmark in the list. Switching to a custom provider automatically unloads WebLLM to free 1-2 GB VRAM.

---

☁️ Cloudflare Workers Deployment

This project uses Cloudflare's modern Workers + Static Assets architecture (replacing legacy Pages) for lightning-fast SSR hydration and duplex WebSocket TTS streaming.

Option A: One-command CLI Deploy (Recommended)

# Builds production bundle and deploys directly to Cloudflare edge
pnpm deploy
# Equivalent to: pnpm build && wrangler deploy

If not logged in, Wrangler will automatically open your browser for OAuth authentication. Once deployed, attach your custom domain (e.g. xiaochun.firetable.tech) under Worker -> Domains & Routes.

Option B: GitHub CI Deployment (Workers Builds)

To deploy automatically on every git push:

1. Log in to Cloudflare Dashboard -> navigate to Workers & Pages -> Create Application -> Workers tab;
2. Select Connect to Git and connect this repository;
3. Build settings (⚠️ Critical):
   • Root directory: Leave completely blank (do NOT enter /) — entering / points the runner to the Linux root filesystem and causes builds to hang;
   • Build command: pnpm build
   • Deploy command: npx wrangler deploy
4. Pushing new commits to main will now trigger automatic build and deployment.

---

📁 Project Structure

Project-XiaoChun/
├── .github/                   # Continuous Integration & Delivery
│   └── workflows/
│       └── release-tauri.yml  # Multi-platform (macOS/Win/Linux) matrix build & GitHub release workflow
├── public/                    # Static assets (hosted via Cloudflare Workers Assets)
│   ├── vrm/                   # Modular VRM base and delta clothing addons
│   │   ├── xiaochun_base.vrmbase  # Stripped base model (~5.9 MB, oxipng compressed textures)
│   │   ├── xiaochun_default.vrmaddon   # Default Techwear outfit
│   │   ├── xiaochun_cheongsam.vrmaddon  # Traditional Cheongsam
│   │   ├── xiaochun_bikini.vrmaddon    # Beach Bikini
│   │   ├── xiaochun_maid.vrmaddon      # Classic Maid dress
│   │   ├── xiaochun_swimsuit.vrmaddon  # Summer Swimsuit
│   │   ├── xiaochun_dinner_dress.vrmaddon  # Dinner Dress
│   │   ├── xiaochun_office_lady.vrmaddon   # Office Lady
│   │   ├── xiaochun_techwear.vrmaddon      # Techwear
│   │   ├── xiaochun_wedding.vrmaddon       # Wedding Dress (13 MB)
│   │   └── xiaochun_shroud.vrmaddon       # Bandage Wrap (with per-outfit bodyMorph baseline)
│   ├── thinking.vrma          # Idle thinking animation loop
│   ├── materials/             # Loading screen key visual (xiaochun_keyvisual*.webp)
│   ├── onnx/                  # EMAGE body motion model weights (vq_*, emage_step, postprocess)
│   ├── _headers               # Cache-Control and security headers
│   ├── robots.txt / sitemap.* # Search engine crawler contracts
│   ├── llms.txt / llms-full.* # AI agent & LLM GEO documentation specs
│   ├── og.jpg                 # OpenGraph share card
│   └── logo.png / favicon.*   # Brand and icon assets
├── src-tauri/                 # Tauri 2.x native desktop hybrid application layer
│   ├── src/lib.rs             # 60Hz alpha bitmask click-through, single-instance IPC, deep-link routing
│   ├── capabilities/          # Desktop permission capability declarations
│   └── tauri.conf.json        # Desktop window, transparent compositing & URL scheme configs
├── docs/                      # Architecture and module deep-dive docs
│   ├── README.md              # Documentation index and agent routing guide
│   ├── INTERACTION_AND_CONTROLS.md # User interaction, shortcuts, mouse gestures & platform differences
│   ├── HYBRID_DESKTOP_APP.md  # Desktop companion mode, click-through, and Tauri 2.x setup
│   ├── PROTOCOL.md            # Inter-process protocol (xiaochun://) specification & actions
│   ├── AGENT_SKILL.md         # Autonomous AI Agent skill specification for companion control
│   ├── INTERACTION_AND_3D_GUIDES.md # 3D holographic guides, wind dynamics & quaternion fixes
│   ├── ARCHITECTURE_AND_RULES.md # System architecture, 10-step lifecycle & developer rules
│   ├── OUTFIT_SWAP.md         # 0-frame T-pose state snapshot, delta addons & IDB caching
│   ├── POSTFX.md              # UnrealBloomPass halos, background bypass & tone grading
│   ├── VRM_BUILD_WORKFLOW.md  # VRM build toolchain, bspatch delta splitting & byte idempotency
│   ├── VRM_ENGINE_AND_WORKER.md # VRMEngine facade coordinator & vrmWorker binary synthesis
│   ├── MOTION_PIPELINE.md     # Universal motion blending & quintic smootherstep inbetweening
│   ├── BODY_TURN_AND_GAZE.md  # Stepping locomotion state machine & saccadic gaze
│   ├── BONE_MORPH.md          # 28-parameter orthogonal biomechanical morphing whitepaper
│   ├── FOOT_IK.md             # Ground anchoring & foot inverse kinematics solver
│   ├── CHAT_DIRECTOR.md       # LLM + TTS + EMAGE streaming orchestration
│   ├── STT.md                   # SenseVoice ChatBar dictation (VAD + ORT Worker)
│   ├── ONBOARDING.md          # Galgame-style immersive onboarding system specification & schema
│   ├── EMAGE_MODEL.md         # EMAGE status & known limits (wasm/INT8, streaming, isolation)
│   └── PERFORMANCE.md         # Performance & size notes: measured numbers, rejected options, unmeasured items (Chinese)
├── packages/                  # pnpm workspace packages
│   ├── project-xiaochun/      # Embed SDK (@firetable/project-xiaochun, published to npm)
│   ├── three-vrm-materials-mtoon/ # MToon fork with NPR shading controls
│   └── emage-onnx/       # Private offline toolchain: slim, quantize and optimize the EMAGE ONNX set, download 191 → 72 MB (not published)
├── scripts/                   # Build, offline asset processing, and protocol runners
│   ├── build-vrm/             # VRM delta packaging and texture optimization (workflow.mjs)
│   └── protocol-call.mjs      # Fast CLI runner for protocol speech testing
├── wrangler.jsonc             # Cloudflare Workers declarative configuration
├── src/
│   ├── routes/                # TanStack Start file-based routes
│   │   ├── __root.tsx         # Root layout (i18n SSR hydration, GEO JSON-LD & meta tags)
│   │   └── index.tsx          # Main index route
│   ├── components/            # React UI components
│   │   ├── onboarding/        # Immersive Galgame-style onboarding tour
│   │   │   ├── OnboardingTour.tsx     # Tour orchestrator, camera checklist gating (4 tasks web / 5 on Tauri) & reply cards
│   │   │   └── onboardingSchema.ts    # Finite-state machine schema, speech lines & branching options
│   │   ├── TauriWindowFrame.tsx   # Native desktop frameless resize handles & corner glow
│   │   ├── TauriTopHeader.tsx     # Desktop ⋯ menu (updates / reload / quit)
│   │   ├── TopHeader.tsx          # Responsive top navigation (scene / wardrobe / lang / settings)
│   │   ├── ChatBar.tsx            # Streaming chat interface & model switcher
│   │   ├── HeadBubble.tsx         # 3D projected speech dialogue bubbles
│   │   ├── dev-drawer/            # Schema-driven 8-section debug drawer (localhost only)
│   │   ├── SliderWithAnchors.tsx  # Optimized slider with onTick / onCommit split
│   │   └── ui/                    # Radix UI primitives
│   ├── hooks/                 # Reusable UI & platform hooks
│   │   ├── usePetUiVisibility.ts  # Click-to-toggle desk-pet UI; hover/dialog holds auto-hide
│   │   └── useDeferredUnmount.ts  # Optional delayed unmount (pet chrome uses opacity instead)
│   ├── core/                  # 3D rendering & scene core (Decoupled Facade architecture)
│   │   ├── vrmEngine.ts       # Central engine coordinator (slim Facade, render loop, VRM loading)
│   │   ├── protocol/          # Inter-process protocol layer (types.ts, handler.ts, index.ts)
│   │   ├── interaction/       # 3D holographic controls (interactionController, turnGuide, pitchGuide, cameraYGuide)
│   │   ├── wind/              # Aerodynamic localized cursor wind forces (windForce.ts)
│   │   ├── scene/             # Scene manager, lineworkWorld, beach3d, voidWorld, and passthroughManager
│   │   ├── lighting/          # 3-channel studio lighting (studioLighting.ts: dir / hemi / fill)
│   │   ├── postfx/            # Post-processing pipeline (postFxPipeline.ts: Bloom/ToneMapping/ColorGrading)
│   │   ├── material/          # MToon material management (vrmMaterialManager.ts: saturation injection)
│   │   ├── morph/             # 28-parameter bone-by-bone scale bindings (vrmBodyMorph.ts)
│   │   ├── outfitSwap.ts      # Atomic outfit swapping (pose snapshot capture & in-memory pre-restore)
│   │   ├── vrmWorker.ts       # Background worker (WASM bspatch assembly & 2-tier IDB cache)
│   │   └── ui/                # Spatial UI tracker (bubbleTracker.ts: raw crown → 2D bubble)
│   ├── motion/                # Motion system (plugin pipeline + bio-inspired posture)
│   │   ├── pipeline/          # tick / selectLiveMotionSource / PoseBuffer / transition
│   │   ├── sources/           # idle, vrma, emage (+ worker), clip, speakIdle
│   │   ├── constraints/       # footIK, bodyTurn, gaze (on draft, then compose)
│   │   └── vrmaRetarget.ts    # VRMA bone retargeting & normalization
│   ├── memory/                # Client-side multi-tier memory (IndexedDB / entity extraction / n-gram retrieval)
│   ├── llm/                   # LLM layer — WebLLM + custom OpenAI-compatible provider factory
│   ├── director/
│   │   └── chatDirector.ts    # LLM → TTS → EMAGE streaming coordinator pipeline
│   ├── i18n/                  # zh-CN / en / ja translation dictionaries + server cookie helper
│   ├── lib/                   # Cross-cutting utilities
│   │   ├── platform.ts        # Desktop/Tauri/Browser environment detection & window actions
│   │   ├── utils.ts           # Shared class merge and speech chunk slicer
│   │   ├── constants.ts
│   │   └── edge-tts-core.ts   # Native Edge-TTS WebSocket client (no third-party TTS SDK)
│   ├── styles/
│   │   └── main.css           # Tailwind v4 @theme tokens + liquid-glass styles
│   ├── App.tsx                # Main application component & event bindings
│   ├── client.tsx             # Client hydration entry
│   ├── server.ts              # Cloudflare Worker entry (SSR + /api/tts WS + COOP/COEP isolation headers)
│   ├── router.tsx             # TanStack Router factory
│   ├── routeTree.gen.ts       # Auto-generated type-safe route tree
│   └── config.ts              # SSOT (R2 / camera / lights / postfx / LLM / bodyMorph / emage.models + emage.motion)
├── vite.config.ts             # Vite 8 + TanStack Start + @cloudflare/vite-plugin
├── vite/                      # Custom vite plugins (extracted from vite.config.ts)
│   ├── localApiPlugin.ts       # /api/tts dev middleware (TTS proxy with EU fallback)
│   └── dropDockerfatAssets.ts # Strip 25 MiB+ WASM blobs for Cloudflare Pages limit
├── AGENTS.md                  # Coding agent engineering standards (motion pipeline + T-Pose / LookAt / state-machine pitfalls)
├── CLAUDE.md                  # Claude-specific runtime context
├── README.md / README-CN.md   # This file (English / 中文)
├── .env.example               # Documented template for all env vars (copy to .env.local)
└── tsconfig.json

---

🌐 i18n & SSR Hydration

Language is stored in a lang cookie and resolved during SSR via @tanstack/react-start/server's getCookie() — wrapped in createServerOnlyFn to keep it out of the client bundle. On the client, the same lang cookie is read via document.cookie, ensuring the SSR-rendered HTML matches the first client paint with zero hydration mismatch.

Switching language:

1. Updates the lang cookie.
2. Calls i18n.changeLanguage().
3. Re-renders all t(...) consumers without reloading.

---

📄 License

This project is licensed under the MIT License.

Third-party assets used by the default scene:

• public/vrm/xiaochun_base.vrmbase and .vrmaddon bundles — VRM character model and wardrobe addons, generated with VRoid Studio (Pixiv Inc.). Subject to the VRoid Studio License — free for personal use, modification, and non-commercial redistribution with attribution. For commercial use, please review the VRoid Studio License terms or contact Pixiv Inc. for a separate agreement.
• public/thinking.vrma — VRM animation. License unknown — verify before redistribution.

/api/tts in src/server.ts (and the Vite middleware in vite/localApiPlugin.ts) uses a hand-rolled native WebSocket client (src/lib/edge-tts-core.ts) to talk to Microsoft's Edge-TTS service directly — no third-party TTS SDK. The TRUSTED_CLIENT_TOKEN constant is a publicly known shared token (the same value used by every open-source Edge-TTS implementation) and is not a personal secret.

---

🙏 Acknowledgements

• pixiv / three-vrm — VRM runtime
• Pixiv / VRoid Studio — default character authoring
• MLC AI / WebLLM — in-browser LLM
• TanStack Start — full-stack React framework
• Tailwind CSS — utility-first styling
• Qiuner / Qiuner.github.io (src/worlds/linework/) — linework outdoor scene visual inspiration
• Animation Inc. — First tried their Ani-2 product and got hooked on the idea of real-time on-device full-body motion synthesis; this project's EMAGE pipeline is our homegrown attempt at the same dream, running entirely in the browser
• PantoMatrix / EMAGE by Haiyang Liu*, Zihao Zhu*, Giorgio Becherini, Yichen Peng, Mingyang Su, You Zhou, Naoya Iwamoto, Bo Zheng and Michael J. Black (*equal contribution), "EMAGE: Towards Unified Holistic Co-Speech Gesture Generation via Expressive Masked Audio Gesture Modeling", CVPR 2024; weights: H-Liu1997/emage_audio (Apache-2.0 per the model card). It is the full-body co-speech motion model that powers our chat-time gesture generation. This project modifies the weights (ONNX export, cross-attention layers removed, INT8 quantization, graph optimization), see packages/emage-onnx/NOTICE.
• VolgaGerm / emage-onnx-export — the PyTorch → ONNX export script and the pre-converted .onnx weights we run in-browser
• snomiao / otoji (otoji.org) — open on-device voice graph; our ChatBar SenseVoice dictation (energy VAD + Worker) learned a great deal from their ?simple path and sherpa-onnx wiring