Clippy Vision

A fully local AI assistant that watches your work to build context automatically without needing to explain much to an LLM. 100% private - no cloud, no data leakage.

---

What is Clippy Vision?

Clippy Vision is a desktop AI companion that passively observes your work - active windows, clipboard, typing patterns, and screenshots - and builds a continuously updating memory of everything you do. When you open the chat, it already knows your context. No copy-pasting. No re-explaining.

Everything runs entirely on your machine. No API keys, no cloud, no data leaving your device.

---

One memory across every app you work in

Your work is not stored in one place. It is spread across the browser, your IDE, local PDFs, terminal output, chat apps, notes files, spreadsheets, and design tools. Each of those keeps its own partial record, or none at all, and none of them know about each other.

Clippy watches all of them and keeps one timeline. Two things follow from that, and neither is possible from any single app's own history:

1. You can search what was on the screen, not just what things were called. Titles and filenames are usually useless later. A paper saved as 2103.00020v1.pdf, a Jira ticket referred to only by its ID, a config you edited in a nameless scratch buffer. Clippy read the content, so the words that were actually in front of you are what you search.
2. You can reconstruct a whole stretch of work, not look up one artifact. "What was I doing Tuesday afternoon" spans the paper you read, the file you edited, the snippet you copied, and the conversation you had about it. Clippy answers that as a summary of the work. Every per-app history hands you a list and leaves the reconstruction to you.

How Clippy Vision fits with Claude / ChatGPT

Claude and ChatGPT are built for reasoning, writing, and general knowledge. They are excellent when you bring them context. They are not built to know what was on your screen yesterday without you telling them.

Clippy Vision is built for the context problem. It watches your work, remembers it, and answers from that memory. It does not replace Claude or ChatGPT. It fills the gap they cannot: your personal activity history.

Use Clippy when you need your own work history back. Use Claude or ChatGPT when you need a strong reasoning partner. Many people use both: Clippy to reconstruct context, then paste that into Claude to go deeper.

One question. Answer pulled from papers, chat tools, and a local notes file from the same research stretch.

Same kind of personal question. Clippy answers from activity it saw on your machine. Claude has no record of that work, because it never saw it.

---

Download

Click your platform to download v1.2.2 directly:

   

All releases & older versions  ·   

The installer includes a setup wizard that handles Python, Ollama, and all required models automatically. No terminal required.

Clippy Vision is under active development. Three releases shipped in the first two weeks, including full macOS support, and bug reports usually get a reply the same day.

Trying Clippy? Share feedback (5 min) to help shape what we build next.

System requirements

Clippy Vision runs a local text model for chat and uses accessibility APIs plus OCR for screen capture (no vision model in the capture path).

• First run needs internet once for the text model (qwen3:8b, ~4.7 GB).
• The setup wizard checks your PC against these numbers before installing. Below minimum → setup is blocked. Between minimum and recommended → you can continue with a warning that chat may feel slower.
• Integrated / shared GPUs are allowed at minimum; a dedicated GPU still helps chat speed.
• Lower-spec / contributor machines: capture uses accessibility + OCR (no vision model in setup). Only the chat model downloads by default; pick a smaller chat model in setup if needed. Details in CONTRIBUTING.md.

---

Quick Start

Option A - Installer (recommended)

1. Use the Download buttons above (Windows, macOS Apple Silicon, or macOS Intel)
2. Follow the setup wizard (installs Python, Ollama, and AI models)
3. Launch from Start Menu → Clippy Vision

Option B - Run from source

git clone https://github.com/protocorn/clippy-vision.git
cd clippy-vision\electron-ui
npm install
npm start

The app will open the setup wizard on first launch and walk you through dependencies.

---

Features

• Passive screen awareness - captures foreground windows, clipboard, typing bursts, and screenshots in the background
• Privacy-first redaction - Clippy Vision's own window is blacked out in every screenshot before the AI ever sees it
• Three-tier event classification - rule-based → feature-based → LLM fallback, so only meaningful events are stored
• Low-cost screen text - accessibility/UI text first with RapidOCR fallback; no vision model in capture
• Hierarchical memory - events → session summaries → distilled long-term facts; memory never resets
• Smart query router - a fine-tuned MiniLM classifier routes every question to the right retrieval strategy before the LLM is even called
• ReAct agent - structured reasoning with tools: SQL generation, memory recall, fact saving
• Conversation memory - rolling summaries + semantic search over past conversations
• Toggle capture - start/stop data capture from the tray icon or the in-app button, with a desktop notification on change
• Per-app redaction (in progress) - backend rules exist for WhatsApp, Telegram, incognito windows, and similar targets; reliable matching outside Clippy's own window is still being improved, so capture on/off is the dependable privacy switch today

---

Where this is going

Clippy is reactive today: you ask, it answers. The next bet is making it proactive, so it can act on what it sees instead of waiting to be asked. Capture now reads window text through accessibility APIs and falls back to local OCR without loading a vision model. A timeline view remains another priority so you can see and delete exactly what was captured.

No dates attached to any of it. PROJECT_VISION.md has the current thinking, the priority order, and an honest list of what does not work yet. If you want to shape any of it, the open issues are the place to start.

---

Tech Stack

---

Architecture

Segment 1 - Data Capture

core/screen_capture.py runs as a background process and captures:

• Active foreground window (title, process name, active URL)
• Clipboard contents (copy and paste events)
• Context switches (window focus changes)
• Keystroke dynamics with per-app adaptive baseline
• Screenshots (taken proactively on activity bursts by core/screenshot_scheduler.py)

Every captured event passes through a three-tier classification pipeline before being stored:

Tier 0 - Rule-based (deterministic, instant) Fast rules that immediately flag obvious signals: too few keystrokes → not interesting; known background system process → not interesting; typing deviation from personal baseline → interesting (score 9).

Tier 1 - Feature-based (scoring) Scoring starts at 5. Multiple features add or subtract: typing deviation, context novelty (how many times this app was seen in 7 days), typing intensity z-score, clipboard content length. Events below 4 are dropped; above 7 are kept; 4-7 go to Tier 2.

Tier 2 - LLM fallback The last 3 events + current event are sent to qwen3:8b for context-aware classification. Output is INTERESTING or NOT_INTERESTING; classification never queues a vision model.

Screen text enrichment Each captured frame records bounded text from the foreground accessibility/UI API. RapidOCR runs only when that text is empty or too sparse. A background processor (core/screenshot_processor.py) groups visually identical screenshots using perceptual hashing and stores the resulting text with the nearest event (±10 s); if none exists, it creates a screenshot_analysis event. Image embeddings and event-level RAG are disabled by default.

---

Segment 2 - Summarization

A background summarizer runs every 5 minutes and groups recent interesting events into session summaries using qwen3:8b. It runs in two passes per tick:

• Pass 1: Summarizes pending events immediately
• Pass 2: Refreshes sessions when delayed screenshot text becomes available

---

Segment 3 - Distiller

Runs every 5 sessions and extracts high-level behavioral facts from summaries. Each fact is:

1. Vector-embedded
2. Compared against existing cluster centroids (threshold: 0.75 cosine similarity)
3. Routed to the closest cluster or a new one
4. Processed with a second LLM call: ADD / UPDATE / NOOP / CONFLICT

Conflicting facts are preserved in memory_conflicts and surfaced to the agent for user resolution. User-provided corrections via save_identity automatically close related conflicts.

---

Segment 4 - Query Router

A fine-tuned MiniLM-L3 classifier (agent/router.py) maps every incoming query to one of:

Each category has a dedicated prefetch module. Context is retrieved in parallel before the LLM is called, so the agent already has relevant data in its prompt without needing to make tool calls reactively.

---

Segment 5 - The Agent

A ReAct agent (agent/react_agent.py) with function calling. Tools available:

Prompt components: conversation history (last 8 turns + rolling summaries), user profile, top-8 memory facts by semantic similarity, and prefetched context from the router.

---

Segment 6 - Database

All data lives in a local SQLite database (core/data/events.db):

FTS5 virtual tables on events and sessions enable full-text search across all stored content.

---

Privacy

• All processing is local. Nothing leaves your machine.
• Clippy Vision's own window is blacked out in screenshots before any AI model sees them.
• You can toggle data capture on/off at any time from the tray icon.
• Per-app redaction is in progress for WhatsApp, Telegram, Signal, incognito windows, and similar targets. Matching is not reliable enough yet outside Clippy's own window, so capture on/off is the dependable privacy switch today.
• Captured data has TTLs: raw events expire after 7 days, session summaries after 90 days.
• The local API binds to 127.0.0.1 on a port chosen at launch, so it is never reachable from your network.

The one outbound request: Clippy Vision checks the public GitHub releases page for a newer version, at most once every 12 hours. It sends no chat, screen, profile, or account data — only the request itself, like opening the releases page in a browser. Turn it off any time under Settings → Updates.

---

Building from Source

# Python dependencies
pip install -r requirements.txt

# Run the desktop app
cd electron-ui
npm install
npm start

# Build the Windows installer
npm run dist

The built installer appears at electron-ui/dist/ClippyVision-Windows-Setup-{version}.exe (or ClippyVision-macOS-{arch}-{version}.dmg when building on macOS).

---

License

MIT - see LICENSE for details.

---

Contributors

Every feature in Clippy Vision has a person behind it. This wall is how we say thank you - by name, with what they actually built, backed by real numbers from git history.

Hall of fame

_{Numbers come straight from git history and refresh automatically on every push to main.}

How to get on this wall

Code is one way in, but not the only one - we follow the All Contributors spec, so a sharp bug report, a design suggestion that sticks, or a doc fix all count: 💻 code · 📦 platform · 📖 doc · 🐛 bug · 🤔 ideas · 🎨 design · ⚠️ test · 👀 review · 🚧 maintenance

When your contribution lands, comment this on the PR or issue and the bot handles the rest:

@all-contributors please add @your-username for code, doc

New here? CONTRIBUTING.md has setup steps and a list of good first issues.

---

Contributing

See CONTRIBUTING.md for setup steps and good first issues, and PROJECT_VISION.md for what the project is optimizing for and where it is headed.