Video Sherlock is an open-source video understanding and local speech toolkit for coding agents. Give it a video URL or local file and it gathers subtitles and metadata, falls back to local speech transcription when needed, finds relevant frames, and produces an auditable report with timestamps, screenshots, and source evidence. The bundled analyze-video skill orchestrates the evidence workflow, the separate synthesize-speech skill handles local narration, and the vq CLI provides their shared local-first engine.

The command scans video frames, ranks keyframes in Rust, embeds selected images with a local Chinese-first image/text model, persists the vectors in SQLite, and searches them with Chinese or English text. It can also transcribe speech from audio or video files completely locally. This is a new, standalone Rust implementation and does not contain the original iOS project's Swift, Objective-C, project files, resources, or copied implementation code.

Primary usage: coding-agent skills

Install it with a prompt

Paste this into your coding agent to install both skills for you:

Install the `analyze-video` and `synthesize-speech` Agent Skills from
https://github.com/jo32/video-sherlock/tree/main/.agents/skills for this coding
agent. Install each complete skill directory in the appropriate user-level or
workspace skills directory. Verify that both skills are discoverable, tell me
their installed paths, and tell me whether I need to start a new session before
using them. Do not analyze a video or synthesize speech yet.

After installation, start a new agent session if requested and use one of the prompts below.

Use it from this repository

This repository ships the analyze-video and synthesize-speech Agent Skills. Clone the repository, then open its root in Codex or another coding agent that supports Agent Skills:

git clone https://github.com/jo32/video-sherlock.git
cd video-sherlock

Ask the agent to use $analyze-video:

Use $analyze-video to analyze https://example.com/video and produce a
comprehensive, evidence-backed Markdown report. Save the analysis under
./video-analyses/example.

For a local file, provide its absolute path and any question you want the report to emphasize:

Use $analyze-video to analyze /absolute/path/to/interview.mp4. Focus on the
speaker's main claims, the evidence shown on screen, and any limitations.

The skill runs the complete workflow: it acquires publisher metadata and timed subtitles, falls back to local speech recognition when needed, indexes the video for Chinese/English semantic frame search, selects and visually inspects useful keyframes, and assembles a final report.md. It also preserves the source and raw evidence in the analysis directory so the result can be audited or resumed.

Ask for standalone local speech with $synthesize-speech without loading the video-analysis workflow:

Use $synthesize-speech to speak "你好，这是本地语音。" and save it as
/absolute/path/speech.wav.

To use the skills from another workspace, copy both complete skill directories into that workspace's .agents/skills directory:

mkdir -p /path/to/your-project/.agents/skills
cp -R .agents/skills/analyze-video /path/to/your-project/.agents/skills/
cp -R .agents/skills/synthesize-speech /path/to/your-project/.agents/skills/

On the first run, analyze-video can install or download supported missing tools, a checksum-verified vq release, and the local models required by active stages. It requires Python 3.10 or newer, FFmpeg, and ffprobe. The separate synthesize-speech skill requires an Apple-silicon Mac and uv. All model downloads are lazy. URL downloads must be content you are authorized to save and remain subject to the source site's terms.

The rest of this README documents installing and using vq directly, including its lower-level indexing, search, keyframe, and transcription commands.

What is implemented

• Streaming, bounded-memory frame analysis for MP4, MOV, MKV, WebM, AVI, MTS, and M2TS files.
• Allocation-free Rust luma scoring using Laplacian variance, gradient energy, exposure, contrast, and clipping penalties.
• Best-frame selection near a timestamp or within fixed time segments.
• Batched, native Rust model inference through Hugging Face Candle.
• Local Chinese/English text-to-image retrieval with OFA-Sys/chinese-clip-vit-base-patch16.
• Persistent SQLite/WAL index with content fingerprints and incremental skips.
• Parallel exact cosine search with Rayon.
• Human-readable output and --json automation output.
• Local multilingual audio/video transcription with timestamped segments.
• Local Chinese-first text-to-speech with Qwen3-TTS on Apple silicon.

FFmpeg and ffprobe are used as the codec boundary. Orchestration, frame quality analysis, visual preprocessing and inference, indexing, ranking, and CLI logic are implemented in Rust. Speech inference is delegated to the separately installed whisper.cpp runtime so Apple Silicon can use its Metal backend. Keeping codec decode in FFmpeg gives reliable hardware/format support without importing any code from the iOS application. Text-to-speech is delegated to a pinned MLX-Audio environment managed by uv, giving Qwen3-TTS access to the Apple silicon GPU without making Python part of the core Rust binary.

Why this embedding model

The default is Chinese-CLIP ViT-B/16: an Apache-2.0, approximately 188M parameter dual encoder trained specifically on Chinese image/text pairs. Its weights are about 753 MB and it runs locally on CPU, so it is considerably more affordable than current 2B+ multimodal embedding models while preserving strong Chinese retrieval behavior.

This is an intentional cost/Chinese-accuracy choice, not a claim that the 2022 checkpoint is the largest or newest model available. Newer models such as Qwen3-VL-Embedding-2B can be stronger but require several times more memory and compute. The index records the exact model identity and refuses to mix vectors from incompatible models.

Why this speech model

The default speech model is SenseVoiceSmall Q8, paired with the 1.7 MB FSMN-VAD. SenseVoice is a Chinese-first, non-autoregressive 234M-parameter model that recognizes Mandarin, Cantonese, English, Japanese, and Korean. It also reports language, emotion, and audio-event tags such as music, applause, laughter, coughing, and sneezing.

The Q8 weights are about 235 MB. The upstream 184-clip Mandarin benchmark reports 8.17% character error for Q8 SenseVoice versus 23.15% for whisper.cpp large-v3-turbo on the same CPU-oriented test. Results depend on the recording domain, but this is a much better cost/accuracy fit for a Chinese-first local CLI.

vq downloads a checksum-pinned native FunASR runtime for Windows x64, macOS ARM64, or Linux x64/ARM64. No Python environment or API key is required. Whisper remains available as an optional engine when timestamped segments, macOS Intel, or languages outside SenseVoice's released five-language checkpoint matter more.

Install a release

Prebuilt vq archives are published for:

• Windows x64
• Linux x64 and ARM64
• macOS Apple Silicon and Intel

Download the archive for your platform from the latest GitHub release, verify it against SHA256SUMS, extract it, and put vq (or vq.exe) on PATH. Each archive also contains the license and third-party notices.

The CLI binary is self-contained, but FFmpeg/ffprobe remain external codec dependencies. Models and the small SenseVoice runtime are verified and downloaded on first use; model weights are not bundled in release archives. The optional vq speak command additionally uses uv and MLX-Audio.

Requirements

• Windows 10/11, Linux, or macOS
• Rust 1.91 or newer only when building from source
• FFmpeg and ffprobe on PATH
• Optional: uv for the project-local yt-dlp video downloader; required for vq speak
• Apple Accelerate is enabled on macOS
• Roughly 1.2 GB free for both default model caches, plus extracted keyframes
• An additional roughly 1.9 GB of cache and about 5 GB of available unified memory for Qwen3-TTS
• Optional: whisper-cli for vq transcribe --engine whisper

On macOS:

brew install ffmpeg rust uv

# Optional timestamped/broader-language transcription:
brew install whisper-cpp

On Windows, install FFmpeg through your preferred package manager and ensure ffmpeg.exe and ffprobe.exe are on PATH. SenseVoice is the Chinese-first default on Windows x64, Linux x64/ARM64, and macOS Apple Silicon. On macOS Intel, install whisper-cli and use the Whisper engine.

Build

git clone https://github.com/jo32/video-sherlock.git
cd video-sherlock
cargo build --release
./target/release/vq doctor

The executable is target/release/vq (target\release\vq.exe on Windows).

Download a video

Install the pinned project tools and download a video with yt-dlp:

uv sync
uv run yt-dlp "VIDEO_URL"

For the best available video and audio merged into an MP4 file:

uv run yt-dlp -f "bv*+ba/b" --merge-output-format mp4 "VIDEO_URL"

Only download content you have permission to save and follow the source site's terms. yt-dlp invokes the separately installed FFmpeg when it needs to merge or convert media.

Direct CLI usage

Check whether all embedding and transcription models/runtimes are already available without accessing the network:

vq model status
vq --json model status

Explicitly prefetch every embedding/transcription model and runtime supported on the current platform. This is an opt-in cache-warming command, not part of the normal CLI or skill workflow. SenseVoice is skipped where no compatible native runtime exists:

vq model fetch

Normal commands download only the model they need at first use: SenseVoice or Whisper for the selected transcription engine, Chinese-CLIP for index/search, and Qwen3-TTS for speak. The component-specific status/prefetch commands are:

# Chinese-CLIP for index/search:
vq model status-embedding
vq model fetch-embedding

vq model status-speech
vq model fetch-speech

# Optional Whisper fallback:
vq model status-whisper
vq model fetch-whisper

# Optional Qwen3-TTS on Apple silicon:
vq model status-tts
vq model fetch-tts

The bundled analyze-video skill never calls broad vq model fetch. Videos with usable subtitles avoid ASR downloads, and --metadata-only avoids the embedding download. Qwen belongs to the separate synthesize-speech skill and is untouched until speech is explicitly requested. prepare_video.py --no-model-fetch enforces cache-only video analysis.

Download state is written to stderr as stable, agent-readable lines. Progress is reported every 5%, while JSON command results remain clean on stdout:

[model] status=missing files=pytorch_model.bin action=download
[model] download-start file=pytorch_model.bin total_bytes=753177983 total=718.3 MiB
[model] download-progress file=pytorch_model.bin percent=25 downloaded_bytes=... total_bytes=753177983
[model] download-complete file=pytorch_model.bin bytes=753177983
[model] status=ready action=continue

The cache honors HF_HOME; otherwise it defaults to ~/.cache/huggingface/hub. Empty or missing files are treated as unavailable.

Transcribe Mandarin, Cantonese, English, Japanese, Korean, or mixed speech from any FFmpeg-readable audio or video file. SenseVoice is the default:

vq transcribe meeting.m4a
vq transcribe interview.mp4 --language zh --metadata
vq transcribe podcast.mp3 --output transcript.txt
vq --json transcribe meeting.m4a --output transcript.json

SenseVoice detects the spoken language; --language acts as an expected-language check and emits a warning on a mismatch. JSON output always contains detected language, emotion, and audio-event fields. Add --metadata to include them in human-readable output.

Use Whisper when segment timestamps or a broader language set is required:

vq transcribe interview.mp4 --engine whisper --language zh --timestamps
vq transcribe lecture.mp3 --engine whisper --prompt "专有名词：滨海新区，抚养权"

SenseVoice does not currently expose segment timestamps through its native runtime. All inference stays on the local machine after the first fetch.

Generate a 24 kHz WAV with the default Chinese Vivian voice on an Apple silicon Mac:

Use $synthesize-speech to turn narration.txt into narration.wav and play it.

Or invoke the underlying CLI directly:

vq speak "你好，我是本地运行的千问语音模型。" --output speech.wav
vq speak "欢迎使用 Video Sherlock。" --output welcome.wav --play
vq speak --text-file narration.txt --output narration.wav

The default model is mlx-community/Qwen3-TTS-12Hz-0.6B-Base-6bit. On first use, uv creates a cached Python 3.12 MLX-Audio tool environment and the model downloads into the Hugging Face cache. Later runs reuse both caches. Use --voice, --language, and --speed to change synthesis settings.

The Qwen Base model can also clone a voice from reference audio. Supply the reference and its exact transcript together:

vq speak "这是克隆声音生成的句子。" \
  --reference-audio reference.wav \
  --reference-text "参考音频里准确说出的文字。" \
  --output cloned.wav

Only use a voice sample when you have the speaker's permission. Qwen TTS is currently available through vq only on Apple silicon; other vq features remain cross-platform.

Find one high-quality frame every ten seconds without loading an ML model:

vq keyframes movie.mp4 --segment-seconds 10 --output-dir ./keyframes

Find the sharpest frame within three seconds of a timestamp:

vq keyframes movie.mp4 --at 60 --radius 3 --output-dir ./keyframes

Index a file or a directory recursively:

vq index ~/Movies --segment-seconds 10

Search in Chinese:

vq search "海边奔跑的一只狗"

Use --json before the subcommand for machine-readable results:

vq --json search "夜晚的城市街道" --limit 20

By default, the index is stored under ~/.video-query. Override it with --index-dir <directory> or the VQ_INDEX_DIR environment variable.

Keyframe algorithm

FFmpeg samples and scales frames to a maximum 640-pixel long edge, then streams 8-bit luma planes over a pipe. Rust evaluates each frame without intermediate image allocations:

focus = sqrt(var(laplacian)) + 0.35 * RMS(sobel-like gradient)
score = focus * exposure_weight * clipping_weight * contrast_weight

The scale cap makes work predictable for 4K/8K input, while fixed-size raw frames make the hot loop cache-friendly. For semantic indexing, the best-scoring frame in each time segment is decoded once at higher resolution and embedded in batches.

Index layout

~/.video-query/
├── index.sqlite3       video metadata, frame scores, and 512-f32 vectors
└── frames/<fingerprint>/
    └── *.jpg           selected frames returned by search

Changing model/schema identity is detected on open. Use a separate index directory after such a change.

Verification

cargo fmt --all -- --check
cargo clippy --all-targets -- -D warnings
cargo test --all-targets

Unit tests cover scoring, padded frame storage, segment selection, vector serialization, model metadata safety, and ranking. The integration test creates a real video with FFmpeg and exercises decode, scoring, selection, and JPEG extraction.

License

This implementation is MIT licensed. Chinese-CLIP is separately distributed under Apache-2.0 and is downloaded from its official Hugging Face repository.