World2Data — Navigation Ground Truth from Video

AI-powered pipeline that converts raw video into structured, navigation-relevant ground truth for humanoid robots. Detects objects, tracks them across frames, classifies navigation states (open/closed/blocked), and exports temporal ground truth with accuracy evaluation.

Quick Start

uv sync
streamlit run app.py

If uv is not installed:

python -m venv .venv
source .venv/bin/activate
pip install -e .
streamlit run app.py

Requires GPU with >=8GB VRAM (sequential offload) or >=16GB (all models loaded).

Features

• Object tracking with persistent IDs across frames
• Zero-shot state classification via SigLIP — 200x faster than VLM for open/closed/ajar labels
• Navigation state classification via VLM (doors, drawers, handles → open/closed/ajar/blocked)
• Temporal diff — VLM compares consecutive frames to detect state transitions
• Navigation timeline — per-object state timeline with colored bars and transition events
• Hand-object interactions via MediaPipe
• Ground truth export — structured JSON with per-frame annotations
• Accuracy evaluation — compare predictions against manual labels
• Live perception — real-time webcam inference with auto-recording and post-analysis
• H.264 video export — browser-playable annotated videos with in-app preview
• Per-frame timing — inference latency breakdown per model stage

Pages

Models

Architecture

core/             — types, ABCs, registry, video I/O
models/           — one file per model variant, registered via @register
  detection/      — yolov8, yolo11, rf-detr
  segmentation/   — sam3, fastsam, rf-detr-seg
  vlm/            — lfm2.5-vl (transformers, onnx, mlx backends in one file)
  tracking/       — yolo-tracker (bytetrack, botsort)
  state/          — siglip zero-shot classifier
pipeline/
  runner.py       — frame-by-frame orchestration, parallel inference, timing
  visualizer.py   — OpenCV drawing for all modalities
  interactions.py — MediaPipe hand-object interaction detection
  nav_state.py    — builds NavigationTimeline from tracker + VLM/state results
  exporter.py     — structured JSON ground truth export
  evaluator.py    — accuracy measurement against manual labels
app.py            — Streamlit UI (upload workflow)
pages/live.py     — Streamlit UI (live webcam workflow)

Adding a Model

1. Create models/<category>/<name>.py
2. Implement the ABC with load(), predict()/update()/classify(), unload()
3. Decorate with @register("category", "name") and import in models/__init__.py

from core.base import Detector
from core.registry import register

@register("detector", "my-detector")
class MyDetector(Detector):
    VALID_SIZES = ("s", "l")  # optional, enables size selector in UI
    def __init__(self, model_size="s"): ...
    def load(self): ...
    def predict(self, frame): ...
    def unload(self): ...

Cluster Deployment (Singularity + SLURM)

Two container images:

# Build image (after dep changes)
singularity build --fakeroot world2data.sif world2data.def
# or TensorRT variant:
singularity build --fakeroot world2data-trt.sif world2data-trt.def

# Deploy
source .env && scp -r . ${CLUSTER_USER}@${CLUSTER_HOST}:${CLUSTER_PATH}

# Submit
sbatch start_app.sbatch        # standard
sbatch start_app_trt.sbatch    # TensorRT

# Tunnel to access UI
ssh -L 8501:localhost:8501 -J ${CLUSTER_USER}@${CLUSTER_HOST} -p 7665 ${CLUSTER_USER}@<compute-node>

Source code is bind-mounted at runtime — no rebuild needed for code changes, only for dependency changes.

Environment

Copy .env.example to .env and fill in:

HF_TOKEN=...           # HuggingFace token (for model downloads)
CLUSTER_HOST=...       # SLURM cluster hostname
CLUSTER_USER=...       # SSH user
CLUSTER_PATH=...       # Remote working directory
SSH_KEY_PATH=...       # SSH key for cluster

See CLAUDE.md for full architecture details, compatibility notes, and development guidelines.