Glance

Glance packages specialized vision-language models for private, offline inference on the phone paired with Meta glasses. It combines:

• an artifact-first model optimization and evaluation platform;
• an optional local MCP control plane for AI agents;
• a runtime-neutral signed .egomodel bundle;
• a reusable Android inference SDK;
• an optional Meta Wearables DAT frame adapter;
• the original fitness coach as the first reference application.

The initial runtime is pinned llama.cpp/mtmd with Qwen3.5-0.8B. LiteRT-LM is the preferred production backend for model families it supports and will be added behind the same SDK API.

Related repositories

Architecture

Model + optional data
        │
        ▼
FastAPI pipeline ──► candidates/evaluation/device profiles ──► .egomodel
                                                               │
Meta glasses POV ──DAT──► Android SDK ◄─────────────────────────┘
                              │
                              └── private offline inference

Repository layout

Quick start

Platform and web UI

First-time setup:

python3.11 -m venv platform/.venv
platform/.venv/bin/pip install -e './platform[test]'
cd web && npm install && cd ..

Start both development servers:

./tools/dev-toolkit.sh

FastAPI: http://localhost:8000
Web UI:  http://localhost:5173

Run the optimization workflow

Open the web UI and follow its six steps:

1. Choose an open-source base model from Glance, upload a fine-tuned Hugging Face checkpoint ZIP/model file, or register a local checkpoint folder. Uploaded checkpoints are inspected without executing model code.
2. Package the selected model unchanged, quantize it for the target device, or select a smaller student for distillation and optional post-distillation quantization.
3. Select an available annotated dataset, upload JSONL/ZIP, or generate open_text, bounding_box, or both labels with images from local diffusion, a Glance-provided API, a custom HTTPS API, or supplied references.
4. Select an Android phone/emulator optimization profile.
5. Review target-aware optimization settings and run the pipeline.
6. Select Android/Kotlin or iOS/Swift, preview the generated integration files, and download the platform-specific .egomodel.

Local diffusion (dataset image generation)

Glance’s wizard Local diffusion model option expects an Automatic1111-compatible /sdapi/v1/txt2img server. The bundled server prefers TensorArt SD 3.5 Medium Turbo GGUF Q4 (sd3.5m_turbo-Q4_K_M.gguf) via stable-diffusion.cpp:

# Public TensorArt weights — HF_TOKEN not required for the default model.
# https://huggingface.co/tensorart/stable-diffusion-3.5-medium-turbo
# First run creates tools/local-diffusion/.venv and downloads GGUF Q4 (~1.8 GB)
# plus text encoders/VAE into .cache/ (gitignored).
./tools/start-local-diffusion.sh

Default URL:

export EGO_LOCAL_DIFFUSION_URL=http://127.0.0.1:7860/sdapi/v1/txt2img

Optionally start diffusion with the toolkit UI:

START_LOCAL_DIFFUSION=1 ./tools/dev-toolkit.sh

See tools/local-diffusion/README.md for VRAM notes, CUDA GGUF build tips, and smoke-test curls. Prefetch weights without serving: ./tools/fetch-local-diffusion-model.sh.

Optional MCP extra

Install the MCP control plane so coding agents can inspect catalogs, submit optimization jobs, and request exports over local stdio:

platform/.venv/bin/pip install -e './platform[mcp]'
EGO_DATA_ROOT="$PWD/.ego-data" platform/.venv/bin/ego-toolkit-mcp

Full setup: docs/mcp.md.

Android SDK and reference app

See docs/android-setup.md.

Native llama.cpp/mtmd .so libraries under android/llama/src/main/jniLibs/ are not committed (~119 MB). Rebuild and copy them with the steps in docs/on-device-vlm.md, or use the published Glance Android SDK.

Copy local.properties.example → local.properties (gitignored) for Android SDK path and Meta DAT github_token.

Verify

export ANDROID_HOME=...   # your Android SDK
./tools/verify.sh

Simulate without glasses

1. Phone camera — in-app “Phone camera (sim)” mode (CameraX).
2. Mock Device Kit — Meta DAT debug path with H.265 video or phone camera. See docs/testing.md.

MCP and Mock Device Kit

• Model Context Protocol (MCP): optional platform[mcp] extra. See docs/mcp.md and the official MCP docs.
• Android SDK: ordinary in-process library — glance-android-sdk.
• Meta Mock Device Kit (MDK): DAT adapter / Android tests to emulate glasses.

Current MVP boundary

The local pipeline provides real validation, deterministic job state, content-addressed packaging, code generation, and bundle verification. Stages that require external quantizers, training infrastructure, paid data generators, or a physical device are explicitly marked simulated or blocked.

See docs/architecture/toolkit-mvp.md.

Meta DAT prerequisites

1. Register at Wearables Developer Center
2. Add a classic GitHub PAT with read:packages as github_token in untracked local.properties
3. Pair Ray-Ban Meta glasses via Meta AI app, or use Mock Device Kit

On-device VLM

Qwen3.5-0.8B GGUF + mmproj run on the phone via llama.cpp JNI. Download weights to the device (not committed to git). See docs/android-setup.md and docs/on-device-vlm.md.

License

Apache 2.0