---

What is ClipCannon?

ClipCannon is an MCP server that turns any AI assistant into a professional video editor. You give it a video file; it analyzes every frame, every word, every emotion, every speaker, every scene -- then exposes 51 tools that let an AI assistant create edits, render platform-ready clips, generate music, clone voices, and produce lip-synced talking-head videos.

The core idea: instead of scrubbing through hours of footage manually, let an AI understand the content through neural embeddings and structured analysis, then have a conversation about what to create.

You: "Find the most emotionally intense moments and create a 60-second TikTok highlight reel"
Claude: [uses clipcannon tools to find moments, create edit, render 1080x1920 clip with captions]

---

Features

• 22-Stage Analysis Pipeline -- Transcription, scene detection, emotion analysis, speaker diarization, narrative structure, beat tracking, OCR, quality scoring, and more. All running as a parallelized DAG.
• 5 Embedding Spaces -- Visual (SigLIP 1152-dim), semantic (Nomic 768-dim), emotion (Wav2Vec2 1024-dim), speaker (WavLM 512-dim), and voice identity (ECAPA-TDNN 2048-dim) embeddings stored in sqlite-vec for KNN search.
• Smart Editing -- Declarative EDL architecture with adaptive captions, face-tracking crop, split-screen, PIP, canvas compositing, motion effects, overlays, and iterative version control.
• 7 Platform Profiles -- One-click rendering for TikTok, Instagram Reels, YouTube Shorts, YouTube Standard, YouTube 4K, Facebook, and LinkedIn with NVENC GPU acceleration.
• AI Audio -- Text-to-music via ACE-Step diffusion, 6 MIDI presets with FluidSynth, 9 DSP sound effects, speech-aware mixing with automatic ducking.
• Voice Cloning -- Qwen3-TTS 1.7B with multi-gate verification (sanity, intelligibility, identity via SECS), best-of-N optimization, and Resemble Enhance post-processing to 44.1kHz broadcast quality.
• Lip-Sync Avatars -- LatentSync 1.6 (ByteDance) diffusion pipeline for talking-head video generation from text scripts.
• Voice Agent ("Jarvis") -- Real-time conversational AI with wake-word activation, streaming ASR, local LLM (Qwen3-14B), and voice-cloned TTS. All local, zero cloud.
• Tamper-Evident Provenance -- SHA-256 hash chain linking every pipeline operation. Every output is traceable to its source.
• Credit Billing -- HMAC-signed balance with Stripe integration, spending limits, and transaction history.
• 100% Local -- No data leaves your machine. All models run on your GPU. All storage is SQLite on disk.

---

Getting Started

Prerequisites

• Python >= 3.12
• FFmpeg (with NVENC support recommended)
• NVIDIA GPU with CUDA support (8+ GB VRAM minimum, 24+ GB recommended)
• An MCP-compatible AI assistant (Claude Desktop, Claude Code, etc.)

Install

# Clone the repository
git clone https://github.com/ChrisRoyse/clipcannon.git
cd clipcannon

# Install core package
pip install -e .

# Install ML dependencies (for GPU analysis pipeline)
pip install -e ".[ml]"

# Install Phase 2 audio/video dependencies
pip install -e ".[phase2]"

# Install dev dependencies
pip install -e ".[dev]"

Connect to Claude

Add ClipCannon as an MCP server in your Claude Desktop config (claude_desktop_config.json):

{
  "mcpServers": {
    "clipcannon": {
      "command": "clipcannon",
      "args": []
    }
  }
}

Or in Claude Code:

claude mcp add clipcannon -- clipcannon

Quick Start

Once connected, tell your AI assistant:

"Create a project from /path/to/my/video.mp4 and analyze it"

The assistant will call clipcannon_project_create followed by clipcannon_ingest, running all 22 analysis stages. After analysis completes (~2-10 minutes depending on video length and GPU):

"Find the best highlights and create a 60-second TikTok clip with captions"

The assistant uses discovery tools to find moments, creates an edit with captions, previews it, and renders a platform-optimized 1080x1920 clip.

Docker

# Build and run with GPU support
cd config
docker compose up -d

# Dashboard available at http://localhost:3200
# License server at http://localhost:3100

---

How It Works

1. Ingest: 22-Stage Analysis DAG

When you analyze a video, ClipCannon runs a directed acyclic graph of 22 stages:

probe -> vfr_normalize -> audio_extract -----> source_separation
                       -> frame_extract -----> visual_embed -> shot_type
                                          |--> ocr, quality, storyboard
                          audio_extract ----> transcribe -> semantic_embed
                                                        -> narrative_llm
                                                        -> profanity
                          audio_extract ----> emotion_embed, reactions, acoustic
                          audio + transcribe -> speaker_embed -> chronemic
                          all signals -------> highlights -> finalize

6 required stages ensure the core data (frames, audio, transcript) is extracted. 16 optional stages run in parallel and degrade gracefully -- if one fails, the rest continue.

2. Multi-Modal Embeddings

Five neural models embed the video content into queryable vector spaces:

All embeddings are stored in sqlite-vec virtual tables for nearest-neighbor search.

3. Cross-Stream Intelligence

Discovery tools combine signals across all embedding spaces to find optimal editing moments:

• Highlight scoring weights emotion (0.25), reactions (0.20), semantic density (0.20), narrative (0.15), visual variety (0.10), quality (0.05), and speaker confidence (0.05)
• Cut point detection finds convergence of silence, beat, scene, and sentence boundaries
• Narrative flow validates that proposed edits tell a coherent story

4. Edit, Render, Publish

The AI assistant creates declarative EDL (Edit Decision List) specifications, previews at 540p for free, iterates with version control, and renders to any of 7 platform profiles with NVENC GPU acceleration.

---

MCP Tools

51 tools organized into 12 categories:

---

Architecture

                    +-----------------+
                    |  AI Assistant   |  (Claude, etc.)
                    |  (MCP Client)   |
                    +--------+--------+
                             | MCP Protocol (stdio)
                    +--------v--------+
                    |  ClipCannon     |
                    |  MCP Server     |  51 tools
                    |  (port: stdio)  |
                    +--------+--------+
                             |
          +------------------+------------------+
          |                  |                  |
  +-------v------+  +-------v------+  +-------v-------+
  | Analysis     |  | Editing      |  | Voice/Avatar  |
  | Pipeline     |  | + Rendering  |  | Engine        |
  | (22 stages)  |  | Engine       |  | (Qwen3-TTS +  |
  |              |  | (FFmpeg +    |  |  LatentSync)  |
  | SigLIP       |  |  NVENC)      |  |               |
  | Nomic Embed  |  |              |  | ECAPA-TDNN    |
  | Wav2Vec2     |  | 7 profiles   |  | verification  |
  | WavLM        |  | ASS captions |  |               |
  | Qwen3-8B     |  | Smart crop   |  | Resemble      |
  | WhisperX     |  | Canvas comp  |  | Enhance       |
  +-------+------+  +-------+------+  +-------+-------+
          |                  |                  |
          +------------------+------------------+
                             |
                    +--------v--------+
                    | SQLite + vec    |  Per-project DB
                    | (analysis.db)   |  4 vector tables
                    +-----------------+  31 core tables

  Separate processes:
  +------------------+  +------------------+  +------------------+
  | License Server   |  | Dashboard        |  | Voice Agent      |
  | (port 3100)      |  | (port 3200)      |  | ("Jarvis")       |
  | HMAC billing     |  | Web UI           |  | Wake word + ASR  |
  | Stripe webhooks  |  | Projects/Credits |  | + LLM + TTS      |
  +------------------+  +------------------+  +------------------+

ML Models Used

Models are loaded on-demand with LRU eviction. GPUs with >16 GB run models concurrently; smaller GPUs load sequentially.

---

Voice Agent

ClipCannon includes a standalone real-time voice assistant:

# Recommended: Pipecat + Ollama (all local)
python -m voiceagent talk --voice boris

# WebSocket server for remote clients
python -m voiceagent serve --port 8765

Lifecycle: DORMANT (CPU only, wake word listening) -> LOADING (~10-20s) -> ACTIVE (full conversation, ~30 GB VRAM) -> DORMANT

Components: Whisper Large v3 ASR, Qwen3-14B FP8 local LLM (~120 tok/s), faster-qwen3-tts 0.6B (~500ms TTFB), Silero VAD, "Hey Jarvis" wake word.

The voice agent pauses other GPU workers on activation and resumes them on deactivation to share VRAM on a single GPU.

---

Database

Each project gets its own SQLite database with:

• 31 core tables -- project metadata, transcripts, scenes, speakers, emotions, topics, highlights, edits, renders, audio assets, scene map, narrative analysis, provenance
• 4 vector tables -- vec_frames (1152-dim), vec_semantic (768-dim), vec_emotion (1024-dim), vec_speakers (512-dim) via sqlite-vec
• Tamper-evident provenance chain -- SHA-256 hash chain linking every operation

---

Configuration

Config stored at ~/.clipcannon/config.json with sensible defaults:

# View all settings
# (via MCP) clipcannon_config_list

# Key settings
processing.whisper_model = "large-v3"       # Whisper model size
processing.frame_extraction_fps = 2         # Frames per second to extract
rendering.use_nvenc = true                  # GPU-accelerated rendering
gpu.device = "cuda:0"                       # GPU device
gpu.max_vram_usage_gb = 24                  # VRAM limit

Auto-detects GPU precision: Blackwell (nvfp4), Ada Lovelace (int8), Ampere (int8), Turing (fp16), CPU (fp32).

---

Credit System

Operations consume credits from an HMAC-signed local balance:

Dev mode starts with 100 credits. Production billing via Stripe webhooks.

---

Project Structure

src/
  clipcannon/           # Core package
    pipeline/           # 22-stage analysis DAG
    editing/            # EDL engine, captions, smart crop
    rendering/          # FFmpeg rendering, 7 profiles
    audio/              # AI music, MIDI, SFX, mixing
    voice/              # Voice cloning + verification
    avatar/             # LatentSync lip-sync
    tools/              # 51 MCP tool definitions
    db/                 # SQLite + sqlite-vec
    gpu/                # Precision detection, model manager
    provenance/         # SHA-256 hash chain
    billing/            # HMAC credits, license client
    dashboard/          # FastAPI web UI
  license_server/       # Credit billing service
  voiceagent/           # Standalone voice assistant
tests/                  # 626 tests across 43 files
config/                 # Default config, Docker Compose
docs/                   # White paper, codestate docs

---

Testing

# Run full test suite (626 tests)
pytest

# Voice agent tests only
pytest tests/voiceagent/

# Integration tests (requires GPU + test video)
pytest tests/integration/

# Lint
ruff check src/

626 tests across 43 files (425 ClipCannon + 201 Voice Agent), plus 10 FSV (Full State Verification) forensic scripts with 750+ individual checks.

---

Supported Formats

Input: mp4, mov, mkv, webm, avi, ts, mts

Output: Platform-optimized mp4 (h264) at 7 resolution/bitrate profiles

---

Documentation

• White Paper -- Full technical paper covering the multi-modal embedding architecture, cross-stream intelligence, and system design
• System Overview -- High-level architecture and tool reference
• Source Code Map -- Complete file tree and dependency graph
• Pipeline Stages -- All 22 stages with models, inputs, outputs
• Database Schema -- Full table definitions and indexes
• Editing Engine -- EDL models, captions, smart crop
• Rendering Engine -- FFmpeg pipeline, encoding profiles
• Audio Engine -- AI music, MIDI, SFX, mixing
• Voice Agent -- Real-time conversational AI architecture

---

Contributing

Contributions are welcome. Please:

1. Fork the repository
2. Create a feature branch (git checkout -b feature/my-feature)
3. Write tests for new functionality
4. Ensure all tests pass (pytest) and lint is clean (ruff check src/)
5. Submit a pull request

---

License

Business Source License 1.1 -- Chris Royse, 2026

You can use, modify, and self-host ClipCannon freely. The one restriction: you cannot use it to offer a competing commercial Video Production Service to third parties. On 2030-03-31 (or 4 years after each version's release), the license automatically converts to Apache 2.0.

---