MiniCPM-V 4.6 (AWQ) — live webcam scene description

Test environment: the browser streams webcam frames to a Python backend, the backend keeps them in a ring buffer and continuously asks openbmb/MiniCPM-V-4.6-AWQ to describe the last 2–3 seconds of video. The description is streamed token-by-token back to the React UI.

browser (React, getUserMedia)  --JPEG @ ~10 fps over WebSocket-->  FastAPI
                                                                     │
                                                    FrameRingBuffer (preallocated numpy, 8 s)
                                                                     │  window(last 3 s)
                                                    MiniCPM-V 4.6 AWQ (transformers ≥ 5.7, gptqmodel)
                                                                     │  streamed text
browser  <--JSON {begin, delta, description, status}--------------  FastAPI

Layout

Setup

scripts/setup.sh          # venv, deps, repo, weights (~1.5 GB)

or by hand:

uv venv --python 3.12 .venv
uv pip install --python .venv/bin/python torch torchvision            # CUDA 13 runtime, driver >= 580
uv pip install --python .venv/bin/python -r backend/requirements.txt
uv pip install --python .venv/bin/python --no-build-isolation gptqmodel  # AWQ kernels for transformers 5.x
cd frontend && npm install

Run

scripts/run_backend.sh              # http://localhost:8000  (model loads in background, ~20 s)
scripts/run_frontend.sh             # http://localhost:5173  (proxies /ws and /api to :8000)

Open the UI → Start camera → Describe continuously. Chrome only allows getUserMedia on localhost or HTTPS; if you open the UI from another machine, use an SSH tunnel (ssh -L 5173:localhost:5173 host).

Without the browser:

.venv/bin/python scripts/smoke_test.py                           # synthetic clip, checks the model
.venv/bin/python backend/describe_source.py --source 0 --stream  # webcam via OpenCV
.venv/bin/python backend/describe_source.py --source clip.mp4 --once
.venv/bin/python scripts/ws_client_test.py [--source 0]           # drive the running backend over WS

How the window is fed to the model

FrameRingBuffer.window(3.0) returns the frames captured during the last 3 s with their timestamps. VideoDescriber stacks them into a (T,H,W,3) array, builds a VideoMetadata(total_num_frames=T, fps, duration) and passes it to processor.apply_chat_template(...) with the parameters recommended in the MiniCPM-V README for video (downsample_mode="16x", max_slice_nums=1, use_image_id=False, stack_frames=3). The processor itself then samples 1 main frame per second plus stack_frames-1 sub-frames per second (composited into a grid), exactly as it would for a video file — so the model sees a "3-second clip", not a bag of images.

Tunables (UI → Settings, or {"type":"config", ...} over the WebSocket):

• window_seconds (1–6, default 3) – how much of the buffer is described
• stack_frames 1–5, downsample_mode 16x/4x – detail vs. speed
• max_new_tokens, temperature, language (en/ru), prompt
• min_interval – extra pause between descriptions (default 0 = back-to-back)

Backend env vars: MODEL_ID (default openbmb/MiniCPM-V-4.6-AWQ), PORT, BUF_WIDTH/BUF_HEIGHT (640×360), BUF_SECONDS (8), ATTN_IMPL, LOAD_MODEL=0 (UI/buffer only, no GPU).

Measured (RTX 5060 Laptop 8 GB, transformers 5.17, torch 2.14+cu130)

Model load ~9 s, warmup (Triton JIT) ~20 s on first start, ~1.9 GB VRAM.

Notes / gotchas found while setting this up

• AWQ backend. transformers 5.x loads AWQ via gptqmodel, which by default wants to JIT-compile its Marlin CUDA kernel (needs nvcc + ninja). Without a CUDA toolkit this fails at load time, so vlm.py passes AwqConfig(backend="gemm_triton") (env AWQ_BACKEND, Triton ships with torch). If you install the CUDA toolkit, AWQ_BACKEND=marlin should be faster.
• stack_frames bug in transformers 5.17 (video_processing_minicpmv4_6.py): the processor builds main-frame + composite "visual units" for the text placeholders but encodes the raw sampled frames, so the model raises Multimodal features and tokens do not match. backend/minicpm_patch.py self-tests for this and swaps in a fixed _preprocess only when needed.
• stack_frames 3 / 4 don't work with this transformers version: the composite of 2 or 3 sub-frames has a different aspect ratio than a frame and the model's vit_merger requires uniform unit sizes in one request. The UI therefore offers 1, 2 and 5 (5 = 2×2 grid, same aspect ratio as a frame).
• nvidia-smi on this machine reports Driver/library version mismatch (kernel module 595.71 vs user-space 595.84) — only NVML is broken, CUDA works. A reboot fixes nvidia-smi.
• Weights are cast to fp16 (AWQ kernels don't support bf16). The AWQ model is the non-thinking variant; the chat template emits an empty <think></think>.
• Camera frames are resized to 640×360 before entering the buffer; the processor resizes again to its own patch grid, so a larger upload only costs bandwidth.
• scripts/ws_client_test.py drives the backend without a browser (synthetic frames or --source 0), useful for latency tests.