Local VLM Bunny

Browser-based video understanding powered by codec-level keyframe extraction and local vision-language model inference. Inspired by the CoPE-VideoLM paper.

Zero backend. Zero API costs. Full privacy.

How it works

Video File (drag & drop)
  |
  MediaBunny (codec-level parsing)
  |
  +-- 'key' packets (I-frames) --> decode --> SmolVLM2 (WebGPU, local)
  |                                           "describe this frame"
  |
  +-- 'delta' packets (P-frames) --> skip or pixel-diff
  |
  Combined descriptions --> video summary

Instead of decoding every frame and running a vision model on each one, we exploit the video codec's own structure:

• I-frames (keyframes): Full images, decoded and analyzed by SmolVLM2
• P-frames (delta frames): Only store differences from previous frames, skipped or cheaply diffed
• This mirrors the CoPE paper's insight: spend compute where it matters, be cheap everywhere else

Stack

Getting started

npm install
npm run dev

Open http://localhost:5173, drop a video file (MP4/WebM), and watch it work.

Requirements

• Browser with WebGPU support (Chrome 113+, Edge 113+)
• First load downloads the model (~500MB), cached in IndexedDB after that

Build & deploy

npm run build

Produces static files in dist/ — deploy anywhere (GitHub Pages, Cloudflare Pages, Vercel, etc).

Architecture

src/
  main.ts           # Orchestrator: drag & drop -> extract -> analyze -> display
  video.ts          # MediaBunny: codec-level keyframe extraction
  diff.ts           # Canvas pixel-diff for delta frame gating
  vision.ts         # Main-thread API for the SmolVLM2 worker
  vision-worker.ts  # Web Worker: SmolVLM2 inference via WebGPU
  types.ts          # Shared TypeScript types
  style.css         # UI styles

Inspired by

CoPE: Efficient Video Language Models via Codec-Based Primitives — the idea that video codec structure (GOP, I-frames, P-frames) can be used to dramatically reduce vision model compute without losing understanding quality.

License

MIT