Gmail Agent MCP

Version: 3.1.0-alpha
Last updated: August 10, 2026
Status: Production-ready autonomous email analysis backend (Node.js + Ollama)
Model: phi4-mini:latest (3.8B, local, free) via Ollama

---

What This Project Is

Most people get too many emails. Some are urgent, some are boring, some contain files or images, some ask you to do something later, and some are only promotional noise.

Gmail Agent MCP is an autonomous email-analysis agent that sits beside your inbox and does the boring parts of email work for you. It reads each email with a local LLM, summarizes it, extracts key points, detects priority/importance, identifies action items, searches related past emails, reads attachments, and can even create calendar events — all without sending your data to a cloud API.

The entire pipeline runs locally (Gmail/Google Calendar API for data access only; all AI inference happens on your machine via Ollama). There are no per-token costs.

---

Table of Contents

1. Tech Stack
2. Architecture
3. How It Works (Pipeline)
4. AI / LLM Layer
5. Vector Search & Embeddings
6. Tools & Capabilities
7. Memory & Learning
8. Autonomous Loop & Scheduling
9. Benchmarks & Validation
10. Setup & Installation
11. Configuration
12. Project Structure
13. Known Limitations & Recent Fixes

---

1. Tech Stack

Core Runtime

AI / LLM

Data & Storage

External Integrations

Document / Attachment Processing

Utilities & Infra

---

2. Architecture

The system is a modular, layered Node.js backend with an autonomous analysis loop, Chrome extension frontend, and an optional Python FastAPI backend.

┌─────────────────────────────────────────────────────────────────┐
│                        Chrome Extension                          │
│  ┌───────────────────────────────────────────────────────────┐  │
│  │ content.js — sidebar UI, email monitoring, WebSocket      │  │
│  │ background.js — service worker, reconnection logic        │  │
│  │ popup.js — connection status, trigger analysis            │  │
│  │ safe-html.js — XSS prevention, DOM escaping               │  │
│  └───────────────────────────┬───────────────────────────────┘  │
└───────────────────────────────┼──────────────────────────────────┘
                                │ ws://localhost:8081
┌───────────────────────────────▼──────────────────────────────────┐
│                      server.js (entry point)                      │
│  - Initialization workflow (fetch → clean → embed → analyze)      │
│  - WebSocket bridge startup                                       │
│  - Periodic analysis scheduler                                    │
│  - AutonomousAgent class with crash recovery                      │
└───────────────────────────────┬──────────────────────────────────┘
                                │
┌───────────────────────────────▼──────────────────────────────────┐
│              server/gmail-server.js (~8.5k lines)                 │
│  - Gmail/Calendar API orchestration                               │
│  - Incremental fetch, analysis batching                           │
│  - Tool execution routing                                         │
│  - Autonomous loop (snooze, follow-up, digest)                    │
│  - FAISS lazy initialization with promise deduplication           │
└──────┬──────────────────────┬────────────────────┬───────────────┘
       │                      │                    │
┌──────▼──────┐    ┌──────────▼─────────┐   ┌──────▼──────────────┐
│  core/      │    │     tools/         │   │    services/        │
│  ai-client  │    │  attachment-tools  │   │ analysis-manager    │
│  ai-models  │    │  calendar-tools    │   │ calendar-manager    │
│  model-mgr  │    │  semantic-search   │   │ core-memory         │
│  constants  │    │  memory-tools      │   │ document-orchestrator│
│  error-hdlr │    │  classification    │   │ query-router        │
│  api-tracker│    │  enrichment        │   │ follow-up-tracker   │
│  resource   │    │  extraction        │   │ smart-snooze        │
│  registry   │    │  action-tools      │   │ daily-digest        │
│             │    │  system-tools      │   │ relationship-graph  │
│             │    │  email-analysis    │   │ sentiment-timeline  │
│             │    │  tool-call-logger  │   │ response-intelligence│
│             │    │  tool-debug-dash   │   │ user-embeddings     │
│             │    │  inline-image-svc  │   │ offline-calendar-queue│
│             │    │  qwen-image-db     │   │ file-reader         │
└─────────────┘    └───────────────────┘   └─────────────────────┘
       │                      │                    │
┌──────▼──────────────────────▼────────────────────▼───────────────┐
│                   Data Layer (SQLite + JSON)                      │
│  gmail_embeddings.db │ documents.db │ email_summaries.json       │
│  core_memory.json │ cleaned_emails.json │ raw_emails.json         │
│  analysis_detail_log.ndjson │ retry_attempts.json                │
└───────────────────────────────────────────────────────────────────┘

┌───────────────────────────────────────────────────────────────────┐
│                   Python Backend (FastAPI)                        │
│  backend/main.py — Uvicorn server, CORS, lifespan                 │
│  backend/services/ — Gmail, AI, Calendar, Vector Search           │
│  backend/agents/autonomous_agent.py — Python port of agent loop   │
│  backend/tools/ — Tool registry + email tool handlers             │
│  backend/api/router.py — Health, email, analysis endpoints        │
└───────────────────────────────────────────────────────────────────┘

Key design principles:

• Local-first: All AI inference is local (Ollama). No email content leaves your machine for analysis.
• Modular: Each concern (AI, tools, services, storage) is isolated in its own module.
• Resilient: Timeouts, retries with backoff, loop detection, and graceful degradation.
• Observable: Every analysis iteration is logged to NDJSON for debugging.
• Dual-backend: Node.js production backend with Python FastAPI migration path.
• Browser-native: Chrome MV3 extension with Shadow DOM isolation and WebSocket communication.

---

3. How It Works (Pipeline)

Initialization (node server.js)

1. Environment validation — checks OAuth credentials, Ollama availability, and required directories.
2. WebSocket bridge — starts on port 8081 for the Chrome extension.
3. Data check — if cleaned_emails.json + gmail_embeddings.db are missing → full init.
4. Fetch — pulls emails from Gmail using incremental history (history.list) when possible, or full mailbox fetch on fresh start. Concurrent fetching with progress bars.
5. Clean — strips HTML/CSS/scripts/comments, removes invisible Unicode chars, normalizes text, truncates to 32K chars for embedding.
6. Attachments — downloads and processes PDFs, Word docs, Excel, PowerPoint, images (OCR + vision captions with moondream), archives, RTF. Stores extracted text in documents.db.
7. Embed — generates BGE-M3 embeddings with MD5 caching, AbortController timeouts, and single-pass top-K selection. Stores in FAISS + SQLite.
8. Analyze — runs the autonomous LLM analysis loop per email with tool calling, loop detection, and crash recovery.
9. Verify — confirms all data files are consistent and initializes background services.

Analysis Loop (per email)

Email → Build prompt (with tools + memory context) → LLM → Parse response
  ├─ Tool call? → Execute tool → Feed result back → LLM again
  └─ Final analysis? → Validate → Save summary → Update vector DB → Next email

The loop runs up to 10 iterations per email, with:

• Loop detection (same tool 3× → stop)
• Parse fallback (YAML → JSON → retry)
• 5-minute timeout per email via Promise.race
• Hallucination filter (removes fake vector-search IDs)
• Tool refresh every 100 emails to keep tool definitions current
• Memory-aware pausing when heap exceeds 512MB, with forced global.gc()

Background Services

• Smart Snooze — checks every 5 minutes for snoozed emails to restore
• Follow-Up Tracker — surfaces follow-ups every 360 minutes
• Daily Digest — summarizes the day's email
• Relationship Graph — sender/recipient relationship mapping over time
• Sentiment Timeline — emotional trend tracking
• Response Intelligence — suggested reply patterns
• Periodic Analysis — incremental analysis of new emails every 5 minutes

Data Flow

Gmail API → Fetch → Clean → Embed (BGE-M3) → Analyze (phi4-mini/qwen3-vl) → Store
                                                              ↓
                                                    Tool Calls (vector_search, calendar, attachments)
                                                              ↓
                                                    Core Memory Learning
                                                              ↓
                                                    Chrome Extension UI

---

4. AI / LLM Layer

core/ai-client.js — Unified AI Interface

• chatOllama() — routes to Ollama (native tools or raw /api/generate)
• _chatPhi4Raw() — bypasses broken Ollama template for phi4-mini
• analyzeEmail() — main entry: builds prompt, runs tool-calling loop, returns structured analysis
• parseToolCallFlexible() — 5-tier parser (FUNCTION_CALL, <|tool_call|>, <tool_call>, JSON, raw)
• extractYAML() / extractJSON() — response parsing with fallback
• _isValidAnalysisJSON() — rejects non-analysis output (tool calls, arrays, empty)

Model Selection

Why phi4-mini? Best small model (<4B) for tool calling (67.1%) and logic/math (88.6%). Proven in production with 0 errors across 83 emails.

---

5. Vector Search & Embeddings

• Embedding model: bge-m3 (1024-dimensional vectors)
• Index: FAISS IndexFlatL2 (exact nearest-neighbor)
• Storage: gmail_embeddings.db (SQLite, embedding as BLOB)
• Search: Temporal-aware — only returns emails before the current email (no future context leakage)
• Self-exclusion: Filters out the email currently being analyzed
• Truncation: Input capped at 32K chars to respect bge-m3's ~8192-token context limit

---

6. Tools & Capabilities

The LLM can call these tools during analysis:

Defensive guards:

• Inline images (IMAGE_* IDs) are already in context — read_attachment rejects them with a clear message.
• Empty-body emails fall back to subject + snippet for embedding.

---

7. Memory & Learning

• Core memory (core_memory.json) — user preferences, patterns, sender history
• Episodic memory (episodic_memory.json) — past interactions
• Relationship graph — sender relationships over time
• Sentiment timeline — emotional trends
• Response intelligence — suggested replies
• User embeddings — personalized semantic profile

---

8. Autonomous Loop & Scheduling

Background services (started at init):

• Smart Snooze — checks every 5 minutes
• Follow-Up Tracker — surfaces follow-ups every 360 minutes
• Daily Digest — summarizes the day's email
• Relationship Graph / Sentiment Timeline / Response Intelligence — initialized at startup
• Periodic Analysis — incremental analysis of new emails (no hard 100-cap)

---

9. Benchmarks & Validation

Production Run — April 18, 2026 (83 emails)

Iteration Efficiency (YAML migration, Apr 18)

Parsing Reliability (JSON → YAML migration)

Tool Usage (83-email run)

Email Category Distribution (83-email run)

• Gaming/Marketing: 42–51%
• Competitions/Hackathons: 18–24%
• Personal/Social: 14–18%
• Work/Professional: 12–15%
• Other: 5–7%

Priority Breakdown

• Low: 57–60% | Medium: 36–40% | High: 1–2%

Performance Optimizations

• Concurrent Gmail fetch (10× speedup vs sequential)
• In-memory cache for cleaned_emails.json (O(1) lookups)
• Batch embedding (sequential to avoid Ollama crashes)
• Prompt truncation (10K-token cap) to fit context window

---

10. Setup & Installation

Prerequisites

• Node.js ≥ 18.17.0
• Ollama installed and running (ollama serve)
• phi4-mini:latest and bge-m3 pulled:

  ollama pull phi4-mini:latest
  ollama pull bge-m3
  

Install

npm install

Authenticate (one-time OAuth)

node authenticate.js

This opens a browser for Gmail + Google Calendar OAuth consent and saves tokens to data/user_tokens.json.

Run (development)

nodemon server.js

Server starts on default port, WebSocket bridge on ws://localhost:8081.

Run (production / Docker)

docker build -t gmail-agent-mcp .
docker run -p 8081:8081 gmail-agent-mcp

Tests

npm test                 # all tests
npm run test:unit        # unit only
npm run test:integration # integration only

---

11. Configuration

Environment Variables (.env)

Key Tunables (core/ai-client.js)

• maxIterations = 10 — max tool-calling loops per email
• ANALYSIS_TIMEOUT_MS = 5 * 60 * 1000 — per-email timeout
• MAX_PROMPT_TOKENS = 10000 — prompt size cap
• TOOL_REFRESH_INTERVAL = 100 — re-fetch tool defs every N emails

---

12. Project Structure

gmail-agent-mcp/
├── server.js                    # Entry point + init workflow + WebSocket
├── server/
│   └── gmail-server.js          # Main orchestration (~8.5k lines)
├── core/
│   ├── ai-client.js             # LLM interface + tool-calling loop
│   ├── ai-models.js             # Model registry (OpenRouter)
│   ├── model-manager.js         # Model init/selection
│   ├── constants.js
│   ├── error-handler.js
│   ├── api-tracker.js
│   └── resource-registry.js
├── tools/
│   ├── attachment-tools.js      # read_attachment, query_document
│   ├── calendar-tools.js
│   ├── semantic-search.js       # FAISS + BGE-M3
│   ├── memory-tools.js
│   ├── classification-rules.js
│   └── ... (10+ tool modules)
├── services/
│   ├── analysis-manager.js
│   ├── calendar-manager.js
│   ├── core-memory.js
│   ├── document-orchestrator.js
│   ├── query-router.js
│   ├── follow-up-tracker.js
│   ├── smart-snooze.js
│   ├── daily-digest.js
│   ├── relationship-graph.js
│   ├── sentiment-timeline.js
│   ├── response-intelligence.js
│   └── ... (20+ services)
├── config/
│   └── prompts.js               # Prompt builders (phi4-mini, Hermes)
├── data/                        # Runtime state (JSON + SQLite)
├── docs/                        # Documentation + benchmark reports
├── extension/                   # Chrome extension frontend
├── tests/                       # Vitest suites
├── scripts/                     # Utility scripts
├── Dockerfile
├── package.json
└── README.md

Note: _legacy_node_backend/ and python_backend/ / backend/ are preserved for reference only. The active code is at the root (server.js, core/, tools/, services/).

---

13. Known Limitations & Recent Fixes

Recent Bug Fixes (Aug 9–10, 2026)

Known Limitations

• Attachment OCR is best-effort; some scanned PDFs may not extract cleanly.
• Vision requires qwen3-vl (optional); phi4-mini is text-only.
• Embedding concurrency is 1 (Ollama crashes with parallel batch embeds).
• Mailbox size tested up to ~100 emails per init cycle; scales via incremental fetch.

---

License & Credits

• LLM: Microsoft Phi-4-mini (local, Ollama)
• Embeddings: BGE-M3 (local, Ollama)
• Gmail/Calendar: Google APIs (OAuth2)
• Vector search: FAISS (Meta)
• Built with Node.js, SQLite, and a lot of patience.

---

Last updated: August 10, 2026 — Documentation refreshed to reflect the active Node.js backend, current model (phi4-mini:latest), and production benchmarks from the April 18, 2026 run (83 emails, 0 errors, 88.7% accuracy).