FUMI

An open-source, Tamagotchi-inspired AI companion for your desk, built on ESP32-S3.

What FUMI Does · Demo · Firmware Setup · Backend · Download Firmware · Contributing

FUMI combines animated expressions, touch interaction, virtual pet mechanics, and real voice conversations in a compact physical device.

It combines:

• animated character UI
• touch interaction
• hunger, energy, happiness, bond, and growth
• a small interactive game
• cloud AI chat
• push-to-talk voice input
• a provider-agnostic AI backend

FUMI is not locked to one model provider. The same firmware can talk to a backend configured for OpenAI, Anthropic Claude, xAI Grok, OpenRouter, Ollama, or another compatible API.

Development status: active / experimental. The firmware in this repository is the latest project source provided for this release. Hardware and provider support should be considered tested only where explicitly marked below.

---

Demo

Architecture

┌──────────────────────┐
│       FUMI           │
│  ESP32-S3 + LVGL     │
│                      │
│  Touch • Life • Voice│
└──────────┬───────────┘
           │ HTTPS
           ▼
┌──────────────────────┐
│     FUMI Backend     │
│                      │
│ /brain     /voice    │
└──────────┬───────────┘
           │
     ┌─────┼──────────────┐
     ▼     ▼              ▼
   OpenAI Claude          Grok
     │     │              │
     └─────┼──────────────┘
           ▼
      OpenRouter
        Ollama
   Compatible APIs

Browser demo

The repository includes demo/index.html, which lets you test /brain from a browser without the physical device. If device authentication is enabled, enter the same optional device token in the demo.

Run the backend:

cd backend
cp .env.example .env
# Configure the provider and model in .env
npm run dev

Then open demo/index.html and enter your backend URL.

Important: the browser demo proves the backend path works. It is not a substitute for an end-to-end hardware demo.

Hardware demo

The repository includes the real prototype demo at docs/assets/fumi-demo.mp4.

Do not publish media that exposes passwords, API keys, private SSIDs, or private conversations.

---

The physical FUMI

▶ Watch the real FUMI prototype demo

Supported AI providers

The backend uses an adapter-style provider configuration.

Set the model ID supported by your account:

LLM_PROVIDER=anthropic
LLM_MODEL=your-supported-claude-model

or:

LLM_PROVIDER=xai
LLM_MODEL=your-supported-grok-model

Model names, availability, pricing, and capabilities change over time. Use the model ID currently supported by your provider account.

Voice architecture

Voice input and text generation are separate.

The reference backend currently uses OpenAI speech-to-text for /voice, then sends the transcript to the selected text model.

For example:

FUMI microphone
      │
      ▼
   WAV upload
      │
      ▼
speech-to-text
      │
      ▼
 transcript
      │
      ▼
selected text model
      │
      ▼
 FUMI response

This means you can use Claude or Grok for FUMI's personality/answer while using a separate speech provider.

---

Repository structure

FUMI/
├── firmware/
├── backend/
├── demo/
├── docs/
│   └── assets/
├── scripts/
├── .github/
├── CONTRIBUTING.md
├── SECURITY.md
├── CHANGELOG.md
├── LICENSE
└── README.md

---

Firmware setup

The firmware requires the upstream Waveshare driver files and the libraries shipped with its Arduino example. Run the dependency setup script first; it fetches the official source and generates the local Arduino sketch folder. Install any bundled libraries listed by the upstream example into your Arduino libraries folder before compiling.

Windows PowerShell:

powershell -ExecutionPolicy Bypass -File .\\scripts\\setup-waveshare.ps1

macOS / Linux:

bash scripts/setup-waveshare.sh

Then open firmware/build/LVGL_Arduino/LVGL_Arduino.ino in Arduino IDE. Edit the generated local file firmware/build/LVGL_Arduino/fumi_config.h with your Wi-Fi credentials and backend URLs.

Arduino IDE

Recommended development baseline:

• Arduino IDE 2.3.10
• ESP32 by Espressif Systems 3.3.12
• Board: ESP32S3 Dev Module
• CPU: 240MHz (WiFi)
• Flash Mode: QIO 80MHz
• Flash Size: 16MB (128Mb)
• Partition Scheme: Huge APP (3MB No OTA/1MB SPIFFS)
• PSRAM: OPI PSRAM
• Arduino Runs On: Core 1
• Events Run On: Core 1
• USB Mode: Hardware CDC and JTAG
• Upload Mode: UART0 / Hardware CDC
• USB CDC On Boot: Disabled

Configuration

Copy:

firmware/fumi_config.h.example

to:

firmware/fumi_config.h

Then fill in:

#define FUMI_WIFI_SSID_CONFIG "YOUR_WIFI_SSID"
#define FUMI_WIFI_PASSWORD_CONFIG "YOUR_WIFI_PASSWORD"
#define FUMI_BRAIN_URL_CONFIG "https://your-backend/brain"
#define FUMI_VOICE_URL_CONFIG "https://your-backend/voice"

fumi_config.h is ignored by Git.

---

Backend

Requires Node.js 20.6+.

cd backend
cp .env.example .env
npm install
npm run check
npm test
npm run dev

Health:

curl http://localhost:8787/health

Chat:

curl -X POST http://localhost:8787/brain   -H "Content-Type: application/json"   -d '{"message":"Hey FUMI, introduce yourself."}'

The backend preserves FUMI's original state-event contract (event, hunger, happiness, energy, bond, affection, playfulness, sleeping, expression, and age_seconds) and returns action, expression, and message. The same /brain route also supports a plain message for the browser demo. The backend returns a simple contract that the firmware can consume:

{
  "ok": true,
  "reply": "Hello!",
  "action": "talk",
  "expression": "normal",
  "provider": "openai",
  "model": "your-model"
}

---

Security

Never put model-provider API keys in the ESP32 firmware.

The intended architecture is:

ESP32
  │
  │ device request
  ▼
Your backend
  │
  │ provider API key stays here
  ▼
AI provider

Before exposing a backend publicly, add authentication/device pairing, rate limiting, usage limits, and restrictive CORS.

See SECURITY.md.

---

Contribute to FUMI

FUMI is an open-source project, and contributions from makers and developers are welcome.

New to the project? Start here:

• Read the contribution guide
• Improve the setup guide — help first-time builders get started.
• Document the hardware setup — help makers understand the components and connections.
• Improve bug reporting — help us make bug reports easier to reproduce.

You can contribute through documentation, reproducible bug reports, firmware improvements, backend development, or UI and animation work.

Before opening a pull request, please read the contribution guidelines and test your changes where possible.

Found a problem or have an idea? Open an issue and tell us what you found.

Project status

FUMI is presented as a complete open-source reference build in this repository:

• animated FUMI UI
• touch interaction
• hunger, energy, happiness, bond, and growth
• mini-game
• Wi-Fi AI backend
• push-to-talk voice pipeline
• selectable text-model providers
• backend device authentication
• multiple speech-to-text provider adapters
• local model support through Ollama / OpenAI-compatible endpoints
• GitHub release automation
• documented Waveshare dependency manifest
• public demo assets and release documentation

Hardware demos and dependency files in this repository should always match the actual hardware build being released.

Download Firmware

Want to try FUMI on your ESP32-S3 board without compiling the firmware yourself?

Get the Latest Build

1. Open FUMI Firmware Build on GitHub Actions.
2. Select the latest successful workflow run on the main branch.
3. Scroll to the Artifacts section at the bottom.
4. Download the fumi-firmware-* artifact ZIP.
5. Extract the ZIP to access the firmware binaries and README.txt.

Target Hardware

• Board: Waveshare ESP32-S3 Touch LCD 1.46
• Flash: 16 MB
• PSRAM: OPI
• ESP32 Arduino core: 3.3.12
• Partition scheme: Huge APP

Important Notes

• These are automated build artifacts and have not necessarily been tested on physical hardware.
• Read the included README.txt before using any binary.
• Verify flash addresses, partition layout, and board settings before flashing.
• Back up your working firmware before replacing it.
• Do not share Wi-Fi passwords, API keys, or access tokens.

Artifact retention: Build artifacts are currently retained for 14 days. Downloaded files can be kept locally.

License

MIT. See LICENSE.

FUMI is an independent open-source project and is not affiliated with or endorsed by OpenAI, Anthropic, xAI, OpenRouter, Ollama, or Waveshare.