JRVS — Self-Hosted AI Operations Assistant

Your team needs AI for the routine work — inbox triage, research digests, document drafting, calendar summaries. But ChatGPT and Claude.ai are off the table. Legal says no. Compliance says no. Your data policy says no.

JRVS is a self-hosted AI operations assistant for teams that need Google Workspace automation with a full audit trail — and can't send that data to OpenAI.

It runs on your servers. Inference happens on your hardware. Nothing leaves your building.

Who It's For

Teams of 10–100 in regulated industries — healthcare, legal, financial services, government contracting — where someone (you) is responsible for keeping AI useful without creating a compliance incident.

You already live in Google Workspace. You have one IT person and Docker. You don't need a six-figure Microsoft Copilot contract. You need an AI that handles the boring-but-sensitive recurring work automatically, keeps a record of what it did, and never touches an external API.

What Makes JRVS Different

1. Nothing leaves your server

JRVS connects to local model backends — Ollama or LM Studio — so inference runs on your hardware. No OpenAI API key. No telemetry. The compliance story is one sentence: the data never left.

2. Autonomous agents with approval tiers

JRVS runs scheduled tasks in the background — morning inbox digest, urgent email watch, weekly research brief — without you being at the keyboard. Each task has a trust tier you configure:

• auto — executes silently (low-risk reads and summaries)
• notify — executes and reports back via Slack or terminal
• confirm — drafts the output and waits for your approval before acting

Nothing sends an email or modifies a document without the tier you set allowing it.

3. Google Workspace integration with a structured audit log

Every write action JRVS takes — sending an email, creating a Doc, updating a Sheet — is logged twice: once before execution (intent) and once after (result), with timestamps and session IDs, in a rotating JSON log on your server. Your compliance team can read and export it.

What It Can Do Today

Licensing

SSO/SAML and multi-user team management are on the roadmap. See ENTERPRISE.md for details.

Full pricing details and FAQ →

Community use is free under CC BY-NC 4.0. Commercial use requires a Professional or Team license.

Quick Start

Prerequisites

Install

# 1. Clone the repo
git clone https://github.com/Xthebuilder/JRVS_Private.git
cd JRVS_Private

# 2. Create a virtual environment (recommended)
python3 -m venv venv
source venv/bin/activate        # macOS/Linux
# venv\Scripts\activate         # Windows

# 3. Install Python dependencies
pip install -r requirements.txt

# 4. Pull a model (Ollama)
ollama pull gemma3:12b

# 5. Run JRVS
python main.py

Automated Setup Scripts

• macOS: chmod +x setup_mac.sh && ./setup_mac.sh
• Windows: setup_windows.bat

Platform Setup

pip install -r requirements.txt
ollama serve
ollama pull gemma3:12b
python main.py

# Install Homebrew if needed
/bin/bash -c "$(curl -fsSL https://raw.githubusercontent.com/Homebrew/install/HEAD/install.sh)"

brew install python@3.11 ollama node
pip3 install -r requirements.txt
ollama serve
ollama pull gemma3:12b
python3 main.py

python3 -m venv venv
source venv/bin/activate
pip install -r requirements.txt
ollama serve &
ollama pull gemma3:12b
python main.py

Usage

Starting JRVS

python main.py                          # Default (Ollama)
python main.py --use-lmstudio           # Use LM Studio
python main.py --theme cyberpunk        # Set theme
python main.py --model gemma3:12b       # Set model (default)
python main.py --debug                  # Debug mode

All options: python main.py --help

CLI Commands

Google Workspace Setup

JRVS can read Gmail, Google Drive, Google Docs, and Google Sheets, and write Docs/Sheets/emails with a full audit log.

1. Create OAuth2 credentials

• Go to console.cloud.google.com
• Create a project → Enable Gmail API, Drive API, Docs API, Sheets API
• OAuth 2.0 credentials → Desktop app → Download as JSON
• Copy the client_id and client_secret values

2. Add to your .env

GOOGLE_CLIENT_ID=your-client-id.apps.googleusercontent.com
GOOGLE_CLIENT_SECRET=your-client-secret

3. Authenticate inside JRVS

/google-auth

Visit the URL printed, grant access, paste the code back. JRVS then starts a background sync that ingests new emails and Drive files into the knowledge base every 15 minutes (configurable with GOOGLE_SYNC_INTERVAL_MINUTES).

4. Use it

/google-status          check connection and sync stats
/gmail search invoice   search Gmail and ingest matches
/gdocs create "Notes" "First draft"
/agent goals            see Gmail-powered autonomous goals
/agent start            run goals on schedule automatically

Extensions

JRVS ships with two standalone extension modules under extensions/. They hook into JRVS memory and events without modifying any core files. Both run entirely locally.

Vision Extension

Connects to any camera — built-in webcam, USB camera, DroidCam over WiFi, or any RTSP/HTTP stream. Captures frames intelligently (only processes frames when motion is detected or on a periodic interval), runs local visual inference, logs observations into JRVS memory, and surfaces anomaly alerts when something unusual is detected.

Quick start:

# Install vision dependencies
pip install -r extensions/vision/requirements.txt

# For LLaVA inference: pull the model into Ollama (already running for JRVS)
ollama pull llava:7b

# Describe what the camera sees right now
python -m extensions.vision.vision_module --describe-once

# Run the full background loop
python -m extensions.vision.vision_module --run

Source options:

Set in extensions/vision/config.yaml:

camera:
  source: "screen"             # or "http://192.168.1.X:4747/video" for DroidCam

Use from JRVS orchestration code:

from extensions.vision import VisionModule

vision = VisionModule()
await vision.initialize()
await vision.start()                            # background loop

description = await vision.describe_frame()     # on-demand snapshot
alerts = await vision.get_alerts()             # anomalies for JRVS to reason about

Once running, ask JRVS naturally:

jarvis❯ What did you see in the last hour?
jarvis❯ Was there any unusual activity at 2am?

Vision observations are embedded and stored in JRVS's FAISS index so they show up in normal RAG retrieval.

Config: extensions/vision/config.yaml

Audio Extension

Full conversational voice I/O. Listens for a wake word ("hey jarvis"), records your speech, transcribes with Whisper, sends the text to JRVS, and speaks the response back using Piper or Kokoro TTS. Everything runs locally.

Install dependencies:

pip install -r extensions/audio/requirements.txt

# Piper binary (Linux x86_64 example)
wget https://github.com/rhasspy/piper/releases/download/2023.11.14-2/piper_linux_x86_64.tar.gz
tar -xzf piper_linux_x86_64.tar.gz
sudo mv piper/piper /usr/local/bin/

# Download a voice model
mkdir -p ~/.local/share/piper-voices
wget -P ~/.local/share/piper-voices \
  https://huggingface.co/rhasspy/piper-voices/resolve/main/en/en_US/lessac/medium/en_US-lessac-medium.onnx
wget -P ~/.local/share/piper-voices \
  https://huggingface.co/rhasspy/piper-voices/resolve/main/en/en_US/lessac/medium/en_US-lessac-medium.onnx.json

Smoke tests:

# Test TTS — hear JRVS speak
python -m extensions.audio.audio_module --say "Hello, I am JRVS."

# Test STT — speak and see transcription
python -m extensions.audio.audio_module --listen-once

# Run full wake-word conversation loop
python -m extensions.audio.audio_module --run

Use from JRVS orchestration code:

from extensions.audio import AudioModule
import aiohttp

audio = AudioModule()
await audio.initialize()

async def jrvs_respond(user_text: str) -> str:
    async with aiohttp.ClientSession() as s:
        resp = await s.post(
            "http://localhost:8010/api/chat",
            json={"message": user_text, "session_id": "voice"}
        )
        data = await resp.json()
        return data["response"]

await audio.start_loop(jrvs_respond)   # say "hey jarvis" to begin

Config: extensions/audio/config.yaml

See docs/EXTENSIONS.md for full documentation on both modules.

How It Works

RAG Pipeline

User query
    │
    ▼
Hybrid Search ──── FAISS (semantic) + FTS5 (keyword) ──── RRF fusion
    │
    ▼
Cross-Encoder Reranker (ms-marco-MiniLM-L-6-v2)
    │
    ▼
MMR Diversity Filter
    │
    ▼
Context injected into LLM prompt → Response

1. Ingestion: URLs/files/vision observations → chunked → BGE embeddings → FAISS + FTS5
2. Retrieval: hybrid FAISS + keyword search → reranked → diverse context
3. Memory: every conversation turn is embedded and stored for cross-session recall

Brave Search Integration

# Set your API key (one time)
export BRAVE_API_KEY="your_key_here"

# Or set at runtime
/brave-key your_key_here

# Search and auto-ingest results
/websearch what is FAISS used for

Results are automatically scraped (full page, not just snippets) and added to your knowledge base.

File Uploads

Drop any text-based file into the uploads/ folder:

cp my_notes.txt /path/to/JRVS/uploads/

Then ingest it:

jarvis❯ /upload ingest-all

Supports 30+ extensions: .txt, .md, .py, .js, .json, .csv, .yaml, .pdf, and more.

Configuration

Environment Variables

All variables can be set in your shell or a .env file.

Example — remote Ollama + DroidCam:

export OLLAMA_BASE_URL="http://192.168.1.100:11434"
export JRVS_VISION_CAMERA_SOURCE="http://192.168.1.50:4747/video"
python main.py

MCP Integration

MCP Client (connect JRVS to external tools)

Configure servers in mcp_gateway/client_config.json:

{
  "mcpServers": {
    "filesystem": {
      "command": "npx",
      "args": ["-y", "@modelcontextprotocol/server-filesystem", "/home/user"]
    },
    "memory": {
      "command": "npx",
      "args": ["-y", "@modelcontextprotocol/server-memory"]
    }
  }
}

JRVS auto-connects on startup. Use /mcp-servers and /mcp-tools to inspect.

Available MCP servers: filesystem, github, postgres, brave-search, memory, slack, and more.

MCP Server (expose JRVS to Claude or other agents)

python mcp_gateway/server.py

17 tools exposed: RAG search, web scraping, calendar, model switching, and more.

See docs/MCP_SETUP.md for full configuration.

UTCP Support

JRVS implements UTCP — a lightweight protocol for AI agents to discover and call tools directly via their native protocols (HTTP, WebSocket, CLI).

# Start the API server
python api/server.py

# Discover available tools
curl http://localhost:8010/utcp

# Call a tool directly
curl -X POST http://localhost:8010/api/chat \
  -H "Content-Type: application/json" \
  -d '{"message": "Hello JRVS!"}'

See docs/UTCP_GUIDE.md for details.

Project Structure

JRVS/
├── main.py                  # Entry point
├── config.py                # All configuration (env vars)
├── requirements.txt
│
├── extensions/              # Plug-and-play extension modules
│   ├── base.py              # BaseExtension ABC
│   ├── vision/              # Vision extension (camera + inference + anomaly)
│   │   ├── vision_module.py # PUBLIC: VisionModule class
│   │   ├── config.yaml      # User config (camera source, model, thresholds)
│   │   ├── camera/          # CameraManager (USB/DroidCam/RTSP), FrameProcessor
│   │   ├── inference/       # LLaVABackend (Ollama), MoondreamBackend (local)
│   │   ├── memory/          # VisionLogger (→ JRVS FAISS), AnomalyDetector
│   │   └── alerts/          # AlertManager (→ JRVS events table)
│   └── audio/               # Audio extension (STT + TTS + wake word)
│       ├── audio_module.py  # PUBLIC: AudioModule class
│       ├── config.yaml      # User config (mic, model, voice, wake word)
│       ├── input/           # MicrophoneCapture, VoiceActivityDetector, WhisperBackend
│       └── output/          # PiperBackend, KokoroBackend, AudioPlayer
│
├── rag/
│   ├── embeddings.py        # BGE-base-en-v1.5 embeddings (768-dim)
│   ├── vector_store.py      # FAISS index (HNSW auto-upgrade at 200k vectors)
│   ├── retriever.py         # Hybrid search + MMR pipeline
│   └── reranker.py          # Cross-encoder reranker
│
├── core/
│   ├── database.py          # SQLite (aiosqlite) + FTS5
│   ├── file_handler.py      # File upload ingestion
│   └── calendar.py          # Calendar events (also used by vision alerts)
│
├── llm/
│   ├── ollama_client.py     # Ollama /api/chat integration
│   └── lmstudio_client.py   # LM Studio integration
│
├── cli/
│   ├── interface.py         # JarvisCLI main loop
│   ├── commands.py          # Command routing
│   └── themes.py            # Matrix / Cyberpunk / Minimal themes
│
├── scraper/
│   ├── web_scraper.py       # BeautifulSoup scraper
│   └── brave_search.py      # Brave Search API client
│
├── api/
│   └── server.py            # FastAPI server + UTCP endpoint
│
├── mcp_gateway/
│   ├── server.py            # MCP server (17 tools)
│   ├── client.py            # MCP client
│   └── client_config.json   # MCP server configuration
│
└── uploads/                 # Drop files here for ingestion

API Integration

from rag.retriever import rag_retriever
from llm.ollama_client import ollama_client

# Add a document to the knowledge base
doc_id = await rag_retriever.add_document(content, title, url)

# Retrieve relevant context
context = await rag_retriever.retrieve_context(query)

# Generate a response
response = await ollama_client.generate(query, context=context)

Troubleshooting

"Cannot connect to Ollama"

ollama serve          # Start Ollama
ollama list           # Verify models are installed
ollama pull gemma3:12b  # Pull a model if list is empty

Or switch to LM Studio: python main.py --use-lmstudio

"Cannot connect to LM Studio"

• Open LM Studio → enable the local server → load a model

Import errors / missing packages

pip install -r requirements.txt
python --version   # Must be 3.8+

Performance issues

• Use a smaller model: ollama pull phi3:mini (for low-RAM devices, or set JRVS_DEVICE_CLASS=sbc)
• Reduce MAX_CONTEXT_LENGTH in config.py
• Clear the vector cache: rm data/faiss_index.*

FAISS dimension mismatch after upgrading JRVS auto-detects and rebuilds the index if the embedding model changed. Old .map files are auto-migrated to SQLite.

MCP servers not connecting

• Install Node.js: node --version and npm --version
• Check paths in mcp_gateway/client_config.json

Vision: "LLaVA model not found"

ollama pull llava:7b

Vision: camera not opening

• Check device index: python -c "import cv2; print(cv2.VideoCapture(0).isOpened())"
• For DroidCam: verify phone and PC are on the same WiFi, open the DroidCam app first

Audio: no sound / mic not detected

# List available audio devices
python -c "from extensions.audio.input.microphone import MicrophoneCapture; MicrophoneCapture.list_devices()"

Set the device name or index in extensions/audio/config.yaml.

Audio: Piper binary not found

• Download from github.com/rhasspy/piper/releases and put piper on your PATH
• Or set the full path in extensions/audio/config.yaml under tts.piper.binary

Contributing

Contributions are welcome, especially:

• New extension modules (vision sources, STT/TTS backends)
• Tool protocol extensions (MCP/UTCP)
• Performance improvements to the RAG pipeline
• Documentation and design feedback

Please open an issue before large changes to align on direction.

License

Creative Commons Attribution-NonCommercial 4.0 International (CC BY-NC 4.0)

Free to use, modify, and share for personal and non-commercial purposes. You must give credit. You may not use this project or derivatives of it for commercial purposes. Respect website terms of service when scraping.

Acknowledgments

Ollama · FAISS · BGE Embeddings · Sentence Transformers · Rich · BeautifulSoup · Brave Search · LLaVA · Moondream · faster-whisper · Piper TTS · Kokoro TTS · openwakeword