🛰️ DepthWizard — ISRO Problem Statement 26175

Single-View Height Estimation and 3D Flythrough

Smart India Hackathon (SIH 2026) · Target: September 10

Implementation status, remaining priorities, reproducible model setup and evaluation requirements: Antigravity handoff. Current system and data-flow diagrams: Architecture. The current CHMv2-in3 checkpoint now has a frozen 30-tile GAMUS AGL evaluation: 2.308 m MAE / 5.190 m RMSE, with 9.114 m building RMSE. These are not absolute DSM accuracy or deployment approval. The earlier ViT-B scores are separate historical results, not this model's accuracy. See completion status and remaining gates and the content-bound evaluation summary.

---

⚡ Quick Start Runbook

1. Backend & Database Setup

The recorded CHMv2 development environment used Python 3.11 and Transformers 5.16.1. Provision an authorized CHMv2-in3 checkpoint and verify the hashes in the handoff before starting this profile. The weights, matching configs, licence and checksums are available in the experimental CHMv2-in3 model release. Local demo catalogues are not included.

# 1. Install backend dependencies
pip install -r backend/requirements.txt

# 2. Initialize only a NEW SQLite catalogue, never an existing saved demo DB
# python backend/db/init_db.py

# Download the model bundle into a new directory using authenticated GitHub CLI.
# Follow checkpoints/serving/MODEL_CARD.md to verify SHA256SUMS before serving.
gh release download model-chmv2-in3-v1 --repo emmanuelmj/depth-wiz --dir checkpoints/chmv2-in3

# 3. Bind the model and its trained seven-class shared head explicitly
$env:DEPTHWIZARD_CKPT = (Resolve-Path "checkpoints/chmv2-in3/chmv2-in3-best.pt").Path
$env:DEPTHWIZARD_MODEL_CONFIG = (Resolve-Path "checkpoints/chmv2-in3/chmv2-config.json").Path
$env:DEPTHWIZARD_JOINT_SEMANTIC_CONFIG = (Resolve-Path "checkpoints/chmv2-in3/joint-semantic-config.json").Path
$env:DEPTHWIZARD_SEMANTIC_PROVIDER = "joint"
$env:DEPTHWIZARD_SEMANTIC_ENABLE_INFERENCE = "1"
$env:DEPTHWIZARD_CPU_THREADS = "4"
$env:HF_HUB_OFFLINE = "1"
$env:TRANSFORMERS_OFFLINE = "1"

# 4. Start the backend, then upload a source or provision a real cached demo
python -m uvicorn backend.main:app --host 127.0.0.1 --port 8010

• Interactive API Swagger Docs: http://127.0.0.1:8010/docs
• Health Check: http://127.0.0.1:8010/api/health
• Model readiness: http://127.0.0.1:8010/api/engine/status (warms the selected model).
• The separate OpenEarthMap provider remains available as an explicit alternative, not a silent fallback for the joint profile.

---

2. Frontend 3D Dashboard (Vite + Three.js + Chart.js)

# 1. Move to frontend directory & install dependencies
cd frontend
npm ci

# 2. Launch Vite dev server
npm run dev

• Open in browser: http://localhost:5180
• The backend is required. API failures are shown instead of silently switching to mock scenes.
• The viewport uses WebGPU first and displays its actual backend. An explicitly labelled WebGL2 path remains available for compatibility and comparison.
• Scene loading starts with a small raster overview, then progressively requests visible height/color tiles. It does not download every full-resolution scene texture at startup.
• The default view is ground, building meshes and separate canopy, not a raised heightfield with object annotations. Supported footprints become actual fitted roof and wall meshes. Their old height relief is replaced with estimated ground, and both assets become visible atomically. Roof planes, courtyard holes and image-aligned placement are preserved. Unsupported regions remain explicitly unmeshed instead of receiving generic template houses.
• Geometry displays the reconstructed shape without RGB imagery. Optical drapes the source image. Height/slope colors are diagnostic layers, never the input to object determination.
• The original prediction can be selected explicitly for comparison. Inspection, exports and accuracy measurements retain that original source rather than the regularized display surface.
• Model setup, licensing, exact classes and limitations: semantic checkpoint guide.

The control panel separates Scene, View and Analysis. Import stays in the header; layers and navigation are immediately available under View. Model details and performance diagnostics are expandable rather than filling the default panel. Below 900px, controls open as a dismissible drawer without shrinking the viewer. Unknown GSD, estimated geometry and visual-only scaling remain labelled.

---

3. Height model (estimated metric AGL)

The backend runs a fine-tuned single-view height model through backend/ml (vendored from the DepthWizard training repo, see scripts/sync_ml.sh). Put a checkpoint in checkpoints/ (see checkpoints/README.md); it loads lazily on the first request, on CUDA if available, else CPU. Latency depends on the selected architecture, image size and machine; do not apply an older ViT-B timing to CHMv2.

Pipeline per upload (POST /api/upload, accepts PNG/JPG/GeoTIFF/GAMUS .h5, optional ?tta=true):

1. AGL - height above ground in metres from the model (sliding 518/512 px windows, Hann-blended).
2. Georeferenced GeoTIFF → Copernicus GLO-30 is fetched for the footprint (public S3, no key), reprojected onto the image grid and fused: GLO-30 supplies everything coarser than 30 m, the model supplies the sub-30 m structure → DSM GeoTIFF in the input CRS (dsm_metric.tif) plus implied terrain (dtm_metric.tif) and agl_metric.tif. Non-georeferenced → rDSM = AGL (rdsm_relative.tif).
3. Viewer assets: optical.png, explicitly encoded height rasters, agl_cm.png, and a revisioned streaming manifest. The frontend samples native heights on the GPU, rather than converting a 16-bit raster through an 8-bit canvas.

/api/inspect and /api/transect sample those stored predictions, not display geometry. Metric units do not establish accuracy against independent truth (absolute: false marks rDSM scenes). /api/benchmarks serves the measured held-out GAMUS test results from backend/static/benchmarks/*.json (produced by the training workflow) - nothing is hardcoded.

📁 Repository Structure

depth-wiz/
├── backend/
│   ├── api/routes.py            # FastAPI REST API endpoints
│   ├── db/
│   │   ├── init_db.py           # SQLite schema creation & seed data
│   │   └── queries.py           # Parameterized SQL query helpers
│   ├── ml/                      # Vendored model wrappers + sliding-window inference (scripts/sync_ml.sh)
│   ├── services/
│   │   ├── height_model.py      # Checkpoint discovery + lazy model loading + AGL prediction
│   │   ├── dem.py               # GeoTIFF metadata, Copernicus GLO-30 fetch/reprojection, GeoTIFF export, slope
│   │   ├── inference.py         # Real height-model pipeline; explicit error when unavailable
│   │   ├── semantic_model.py    # Reviewed local semantic weights or source-bound masks
│   │   ├── semantic_unet.py     # Independent OpenEarthMap U-Net inference adapter
│   │   ├── reconstruction.py    # Class-constrained roof/road/terrain display surface
│   │   ├── calibrator.py        # Affine least-squares & RANSAC helpers
│   │   ├── agl.py               # Morphological ground filter (fallback path only)
│   │   └── exporter.py          # 16-bit PNG & GeoTIFF exporters
│   ├── static/demo_data/        # Precomputed source scene bundles
│   ├── utils/
│   │   └── generate_mock_textures.py
│   ├── main.py                  # FastAPI application entrypoint
│   └── requirements.txt
├── frontend/
│   ├── src/
│   │   ├── 3d/
│   │   │   ├── terrain.js       # WebGPU scene facade, lighting and terrain controls
│   │   │   ├── streaming/       # Shared TSL materials, quadtree LOD, tile cache, data-driven footprints, picking
│   │   │   ├── cameraFlight.js  # Catmull-Rom drone flight spline
│   │   │   └── picking.js       # Raycast UV-to-raster pixel mapper
│   │   ├── chart/profileChart.js# Chart.js 2D transect drawer
│   │   ├── hud/
│   │   │   ├── presets.js       # 4 Preset cards UI
│   │   │   ├── inspector.js     # Floating point inspection HUD badge
│   │   │   └── layers.js        # Optical / Heatmap / Wireframe switcher
│   │   ├── styles/theme.css     # ISRO dark theme (#0B0F19, #00F2FE)
│   │   ├── api.js               # Same-origin API client with explicit errors
│   │   ├── mockData.js          # Legacy fixtures, not the live scene-loading path
│   │   └── main.js              # Viewport orchestration & render loop
│   ├── index.html
│   ├── package.json
│   └── vite.config.js
├── notebooks/
│   └── depth_anything_colab.ipynb # Google Colab T4 inference & validation
├── data/
│   └── sample/DC_03_26_RGB.h5   # Sample satellite RGB tile
├── docs/                        # Complete technical architecture & PRD
├── depth.db                     # SQLite database catalog
├── AGENT_PROMPT.md              # Master agent initialization prompt
└── README.md

---

Rendering and performance

The renderer uses shared 64-segment tile geometry and guttered height/color samples. Fine boundary vertices follow the coarser neighbour's sampled edge, closing LOD cracks without adding vertical skirt walls. A screen-space error selector chooses a non-overlapping quadtree cover within a visible-tile budget. Parents remain visible while child resources are loading. Requests are prioritized, stale requests are cancelled, and an LRU cache disposes evicted textures and derived geometry.

Quality presets select 16/32/64 visible tiles and 32/64/128 MiB tile allocation budgets. These are tracked asset allocation estimates, not measurements of total physical VRAM. Structured terrain contains ground beneath supported buildings; separate roof/wall meshes replace their former height relief. Meshes are clipped to the selected tile cover so they do not overlap between tiles or create walls along internal tile cuts. Optical roof UVs use the same source pixels as the footprint. Walls are neutrally shaded rather than stretching roof imagery into invented facades. Optional object outlines remain annotations, not another solid representation.

The Ground + buildings + tree canopy view draws source-bound foliage above the same ground reference instead of raising terrain into tree-shaped mounds. Only confident tree pixels contribute foliage; estimated stems use supported crown candidates, not randomly placed objects. Leaf clumps, crown thickness and stem diameters are display assumptions, not recovered tree anatomy. The older continuous canopy heightfield remains an explicitly named diagnostic. Current reconstruction uses closed, source-bounded canopy shells with tapered estimated undersides, not overlapping crown skins or open cutout curtains. Stems and branches remain limited to supported candidates and attach within the fitted canopy body. Source predictions are unchanged.

Semantic reconstruction is distinct from the raw diagnostic height view and requires a verified local semantic model or source-aligned cached masks. Estimates are not surveyed building models or recovered hidden facades. Focus a mesh building provides a close inspection of actual geometry.

The same shader variants are reused as inputs change. Height colors interpolate the same vertex heights as the rendered triangles; they do not resample a different bilinear surface. Height and slope colors are unlit and unaffected by exposure. Geometry shading and slope use the rendered triangle normals, including stitched edges, on both WebGPU and WebGL2. Shading layers do not evaluate every visualization mode at once. Shadows are optional, and static scenes stop submitting renders. Ground/display height is labelled separately from the original source values used for inspection and export.

Building display height defaults to 1x above the fitted foundation, with a separate control from overall relief. Shape-preserving architectural cleanup only flattens near-planar noise within a bounded source-deviation tolerance. Supported ridges, valleys and roof wings remain; this reversible display approximation does not alter model metres, source maps, terrain or tree heights.

Source prediction map (2D) reads the original float32 predictions separately from 3D geometry. Ground filtering and object replacement do not zero its values, and the heatmap does not generate terrain. Unknown-GSD metre inputs use a labelled pixel-relative display scale rather than automatic range exaggeration; optical-only uploads can supply their known GSD. Real broad terrain still comes from an aligned terrain reference, not from treating AGL tree/building heights as hills.

Reviewed RGB identities can restore published dataset resolution when an input has lost its metadata. backend/static/source_catalogue.json contains only explicitly verified pixel hashes and provenance, not filename guesses or height references. The verified NYC_00918 image uses the GAMUS paper's published 0.33 m resolution. Resized, altered and unrecognized images remain unknown.

Scene selection, model identity and display state are visible in the workspace. Inspection distinguishes displayed geometry from original AGL/surface values. Analysis can download the original raster or plot a cross-section between two chosen pixels on the source map. Recorded metrics are selected by report/model, not ranked across different reference selections. UI and navigation listeners are released when the viewer is disposed.

Use the performance display/profile action for device measurements. Headless runs on SwiftShader measure software rendering, not an RTX GPU, and are not an FPS guarantee.

cd frontend
npm test
npm run test:browser  # requires the Vite server on :5180 and Playwright Chromium
npm run test:live     # actual app + backend, source/structured comparison and persisted telemetry
npm run test:workflow # actual source-map, inspection, profile, export and navigation workflow

Browser regressions use generated fixtures through intercepted local routes, not remote inference services. Reports and snapshots stay in ignored frontend/reports/. See the frontend implementation guide for coordinates, resource ownership and the streaming contract.

Shadow-robust model preparation

The height transformer and the semantic U-Net need separate training. GAMUS supplies real class masks as well as heights; use the canonical data exporter's --with-classes option instead of deriving semantic labels from height maps.

The training guide covers localized RGB-only shadows, supervised/boundary/paired-consistency losses, clean/shadow evaluation, and content-bound promotion gates. These are preparation tools: no running job or deployed checkpoint changes automatically. New candidates stay unapproved until they pass supervised held-out evaluation without clean-image regression.