Floaty 🛟

A local-first AI assistant that floats on top of your desktop.

---

Floaty lives in your tray / menubar and as a draggable swimming-ring overlay that stays on top of your other windows. Through the ring you can capture what's on your screen - a screenshot plus the actual text content read via accessibility APIs - and everything you capture is embedded into a local memory that the assistant can search when you chat with it.

Everything stays on your machine: memory, conversations, skills, settings, and speech-to-text all live in ~/.floaty. The only thing that leaves your computer is what you send to the LLM provider you configure - and with on-device embeddings plus a local model in Ollama, nothing has to.

✨ Features

• Floating ring overlay - draggable (with natural ring rotation), borderless, always on top, and click-through in its transparent regions. Buttons for screenshot capture, voice input, chat, and settings; the ring image itself is customizable. Capturing snaps the ring shut like a camera shutter.
• Sound effects 🔊 - a shutter sound on capture and a chime when a reply finishes, each toggleable and swappable for a built-in or your own file in ~/.floaty/sounds.
• Screen capture & reading - grabs a screenshot and the text content of the active window via UI Automation, so captures are searchable by meaning, not just stored as pixels.
• Automatic screen history - optionally records the foreground window (title and/or content) as you work, feeding your local memory without any manual capturing.
• Written to be read by an LLM - in text-only mode the day's activity becomes a single markdown file of time-stamped blocks. Interface junk is filtered out, each line is written once per day no matter how often it's on screen, and every block records the URL or document path it was looking at so your assistant can open the real thing instead of trusting a fragment.
• Redacted before it's written - password managers and private-browsing windows are never captured, password fields are skipped at the source, and credentials, API keys and card-shaped numbers are scrubbed before anything touches disk.
• Local memory with vector search - captures are embedded and stored in an on-device LiteGraph/SQLite database. The chat exposes a search_captures tool so the assistant can recall what you've seen.
• Any model provider 🔌 - configure OpenAI, Anthropic, Gemini, Azure OpenAI, OpenRouter, Groq, Mistral, DeepSeek, xAI, Ollama, or any OpenAI-compatible endpoint side by side, each on its own tab. Chat, embeddings and screenshot captioning are assigned independently, so you can mix providers.
• On-device embeddings 💻 - download a small ONNX sentence-transformer (BGE, MiniLM) and Floaty embeds every capture in-process. Screen history runs constantly, so this is the difference between it costing something per window switch and costing nothing - and it keeps working with no API key at all. Pair it with a local vision model in Ollama for fully offline memory.
• Chat interface - conversations with the configured LLM, saved locally as JSON. A user-editable system prompt lives at ~/.floaty/floaty.md.
• Voice input 🎤 - fully local speech-to-text: Silero VAD segments your speech and transcribe.cpp transcribes it with GGUF models (Whisper, Voxtral, …) downloaded on demand.
• Agent skills - drop SKILL.md-based skills into ~/.floaty/skills (also picks up ~/.claude/skills and ~/.agents/skills) and invoke them with /name in chat.
• MCP support - connect Model Context Protocol servers and scope the chat to a server's tools with a /server slash command.
• Auto-updates - the Windows build updates itself in place via Velopack, fed from GitHub Releases.

📥 Installation

Grab the latest Windows installer from the Releases page. The app checks for updates itself after that.

On first run, open Settings > Model Provider and add a provider. OpenAI, Anthropic, Google Gemini, Azure OpenAI, OpenRouter, Groq, Mistral, DeepSeek and xAI have one-click presets; Ollama, LM Studio, llama.cpp's server or any other OpenAI-compatible endpoint work through the Ollama and Custom entries.

Platform support: Windows is the primary platform with the full feature set. A Mac Catalyst target exists with the overlay working and other native features (screen capture, screen history, voice input, autostart) stubbed out for now.

📁 Local-first layout

Everything Floaty knows lives under ~/.floaty:

🛠️ Building from source

Prerequisites:

• .NET 10 SDK (app/global.json pins 10.0.301, rolling forward on feature bands)

No workloads to install.

cd app
dotnet build
dotnet run

Floaty currently builds for Windows only. macOS support is planned but not wired up: see the "macOS" note in AGENTS.md for what it needs.

The solution file is app/Floaty.slnx if you prefer Visual Studio.

🧱 Tech stack

• Avalonia 12 — all-native UI, no embedded browser anywhere
• Microsoft.Extensions.AI for chat, tools, and embeddings, over OpenAI / Azure / Anthropic / any OpenAI-compatible endpoint
• LiteGraph (SQLite) for local vector + graph memory
• ModelContextProtocol for MCP client support
• Win32 interop, UI Automation, and GDI for the Windows overlay and screen capture
• NAudio + ONNX Runtime (Silero VAD) + transcribe.cpp for local voice input
• Velopack for packaging and auto-updates

📂 Repository layout

app/       Avalonia application (Floaty.slnx, Floaty.csproj)
website/   Landing page deployed to GitHub Pages
.github/   CI: Windows release packaging + Pages deployment

See AGENTS.md for a deeper architectural guide (aimed at coding agents, useful for humans too).