Recall

Ask a first-person recording what happened and where an object was last seen. Local FastAPI + SQLite + FFmpeg + Grounding DINO Tiny + RelateAnything.

Current state — 2026-09-15

The Mac prototype is built. Three golden queries pass against an explicitly synthetic, 32-second fixture. Both actual ML models load and run locally, including with model-hub networking disabled. Carl's controlled footage has not arrived; perception accuracy and the real three-query demo are not validated. Modalix has not been integrated.

This is separate from ~/CODELocalProjects/recall, Carl's existing life-memory index. It does not change LifeOS, Mentra firmware, phone settings, capture services, or task vaults. Mentra Live and Samsung S26 were both visible over USB/ADB on 2026-09-15.

Run

Python 3.12 (RelateAnything currently requires 3.12+, correcting the brief's 3.11). FFmpeg/ffprobe must be on PATH. Dependencies and upstream code are locked in uv.lock; model revisions are pinned in backend/config.py.

cd ~/CODELocalProjects/recall-glasses
uv sync --extra dev --extra ml
# First use downloads the pinned models; the current Mac already has them cached.
HF_HUB_OFFLINE=1 TRANSFORMERS_OFFLINE=1 .venv/bin/python -m uvicorn backend.main:app --host 127.0.0.1 --port 8877

Open http://127.0.0.1:8877. No CDN, remote font, cloud inference, or API key is used. This is a single-user, localhost prototype. Run one server worker per data directory; no durable job queue or production deployment is implemented. Startup marks unfinished jobs interrupted. Import again to create a fresh run; old inputs and debug outputs remain.

Ingest the recordings when available

Add MP4/MOV files through the UI, or:

HF_HUB_OFFLINE=1 TRANSFORMERS_OFFLINE=1 .venv/bin/python -m scripts.process_video /absolute/path/full_demo.mp4
# If genuinely known, include --start-time 2026-09-15T15:00:00-07:00

An unknown capture time stays unknown. Results use a video-relative timestamp; upload or file-modification time is never substituted. Separate recordings can only be ordered across each other if capture times are known. A single sequential full-demo recording avoids that ambiguity. Individual recordings can be selected in the UI.

The processor accepts up to 10 minutes / 4K, and the HTTP upload accepts up to 1 GB. Use the brief's 720p/1080p, 3–5 minute controlled sequence first. Sampling is 2 Hz using actual decoded presentation timestamps, not assumptions about constant source FPS. The default selects Apple GPU (MPS) when available, otherwise CPU. Unsupported MPS setup/inference falls back to CPU with a warning. RECALL_DEVICE=cpu forces CPU. This is preprocessing; performance on actual recordings remains unverified.

History: the initial prototype defaulted to CPU. On 2026-09-15, five repeated-image runs measured warm detector+relation times of about 0.4–0.5 seconds on MPS versus 5–6 seconds on CPU, so the default changed to automatic selection. Cold setup was slower, and this small calibration test is not a video throughput benchmark.

Queries

• What was I doing before I started coding?
• When was I fixing my bike?
• Where did I leave the wrench after fixing my bike?

Also supports last-seen queries for bottle, phone, and backpack. Intent parsing is limited and deterministic; unsupported questions return a supported-query hint.

Synthetic walkthrough (isolated from personal recordings)

.venv/bin/python -m scripts.build_fixture --data data/fixture
RECALL_GLASSES_DATA="$PWD/data/fixture" .venv/bin/python -m uvicorn backend.main:app --host 127.0.0.1 --port 8878

The current fixture already exists; do not rebuild into the same directory. Its graphics and observations are scripted and visibly labeled synthetic test data. It exercises real video decoding, tracking, temporal logic, SQLite, clips, overlays, API, and UI; it does not test model accuracy. Use port 8877 for personal videos.

Evidence and honesty

• Every answer comes from SQLite observations of a completed ingest.
• Failed/interrupted/partially processed recordings cannot answer questions.
• Location answers say last seen, including when the last sighting is in a hand. A later unknown/held observation supersedes an older spatial location.
• IoU tracks are scoped to one recording, expire after a short gap, and do not perform re-identification. Multiple possible tracks produce an abstention. Fast egocentric camera motion is an important unvalidated failure mode.
• A placement candidate needs a prior holding observation followed by repeated spatial evidence while a hand remains visible. This is a heuristic, not proof of release.
• Activity names are demo rules, inferred from at least 5 seconds of support and persistent observations. A laptop/keyboard interaction does not establish that someone was actually coding; the UI labels this inference.
• Confidence fields are model/rule scores, not calibrated probabilities that an answer is correct. The UI does not display misleading percentage certainty.
• Evidence clips are silent video extracts with 2-second context on either side.
• A reference photo smoke test produced a questionable 0.37 wrench detection near a handlebar. Do not treat successful model execution as evidence of detector accuracy.

Data

data/ is ignored by git. RECALL_GLASSES_DATA selects a separate root.

raw/             original copied recording (never modify the source)
frames/<id>/     sampled JPEGs
clips/<id>/      episode MP4s
overlays/<id>/   object/reference evidence highlights
debug/<id>/     raw model tensors, decoded predictions, retained relations, timings
recall.db        videos, frames, instances, observations, relations, episodes, states

The API exposes only evidence JPEGs and clips. It does not expose the SQLite file, raw videos, logs, or model tensors. The initial app is intentionally local-only. Raw predictions include all scored pairs' logits and the model-returned ranked relation list before application filtering, plus filtered relations at the configured threshold. The application only uses the fixed 11-predicate vocabulary from the brief.

Validation

.venv/bin/pytest -q
.venv/bin/ruff check backend scripts tests
node --check frontend/app.js
# Real image inference, with retained tensors and model metadata:
HF_HUB_OFFLINE=1 TRANSFORMERS_OFFLINE=1 .venv/bin/python -m scripts.model_smoke /absolute/path/frame.jpg

Tests cover the three golden queries, clip/still existence, timestamp honesty, track continuity/expiry, one-to-one assignment, activity gaps, placement evidence, unknown chronology, later observations superseding older locations, failed-ingest visibility, non-zero FFmpeg exits surfacing their stderr instead of passing silently, refusal to invent timestamps when decoded PTS do not match saved frames, API behavior, and media path boundaries. The fixture does not substitute for the planned 20-frame detector/relation check on actual footage.

The /media guard authorizes the resolved path. An earlier version checked the allow-listed prefix on the raw request and containment on the resolved path, so /media/frames/%2e%2e/raw/<video_id>.mp4 served an original personal recording, and GET /api/videos lists the IDs. Fixed on 2026-09-15; the traversal is now a regression test against a real raw file, not only against a .db suffix that failed for other reasons.

Next gate

When the recordings arrive: inspect about 20 representative frames, verify wrench, bicycle, laptop and hands, check relationships first on selected boxes, then on detector boxes. Tune thresholds/prompts from that evidence. Process the complete sequence and verify all three answers against the original frames. Only then proceed to Modalix.

Primary sources and licenses

• RelateAnything upstream, pinned at 6b9c07c12ff47d4be17fa558b183023dcf829e61.
• Released relation model, pinned at f511f17c65451a246997c7c5a932420dee0b0fd3.
• Grounding DINO docs.
• Grounding DINO Tiny, pinned at a2bb814dd30d776dcf7e30523b00659f4f141c71.

Upstream RelateAnything code is Apache-2.0; its checkpoint is subject to the DINOv3 license and upstream third-party notices. No upstream weights are committed here. Public calibration images were supplied by RelateAnything's reel assets: bicycle by Clem Onojeghuo (CC0), cat/laptop by acebrand (CC0). Original credits are retained under models/calibration_credits.json. Those are calibration inputs, not Carl's memories.