SatQuery AI — Satellite Imagery Question Answering

Satellite imagery question answering, remote-sensing change detection, visual grounding and optical-SAR fusion in one open, CPU-runnable system.

Ask a natural-language question about a satellite or aerial image — or a pair of images — and SatQuery AI routes it to the right specialist model, collects evidence, and returns a single confidence-scored result envelope. It is a remote-sensing vision-language system built as a router plus specialists pipeline: satellite image captioning, visual question answering, text-guided visual grounding, bi-temporal satellite image change detection, change-VQA, and Sentinel-1 / Sentinel-2 optical-SAR fusion. It runs on CPU, is served from a static frontend, and is live at https://satquery.pages.dev.

Status: research prototype, pre-1.0. The architecture is frozen. This repository is the documented public release of the system, its trained artifacts, and its measured results — including the negative ones.

This README is the front door to a long-form documentation set. It is written to the same standard as the rest of the release: every number, path, run identifier and status is taken from a file that was actually read, and where something was never run, that is stated rather than implied.

---

Topics

This system spans several searchable areas. If you are looking for a remote-sensing vision-language model, a satellite image change-detection baseline, a text-guided visual grounding implementation for Earth observation, Sentinel-1/Sentinel-2 optical-SAR fusion, or a router-and-specialists design that runs on CPU, this repository covers that ground.

Search terms this project is described by: satellite imagery question answering · remote sensing vision language model · satellite image change detection · visual grounding remote sensing · optical SAR fusion · Sentinel-1 Sentinel-2 fusion · satellite image captioning · LEVIR-CD · VRSBench · BigEarthNet · router and specialists architecture · vision-language model on CPU.

---

Table of contents

• Topics
• Motivation
• What the system supports
• Supported inputs
• Architecture
  • Repository map
  • The four endpoints
  • Documentation map
• Routing and the execution trace
  • The eight execution events
• Real inference vs. the preview path
• Models
• Measured results
  • Grounding: three decode variants, two protocols
  • Fusion: measured but the ruling is open
  • Change-VQA: two test sets, and they disagree
  • Phase 6 / VLM: deployment success ≠ model acceptance
  • Calibration: it got worse, and we say so
• Live validation
  • Representative real run IDs
  • Screenshots
• Installation
• Local development
• Deployment
  • Deployment caveats
• Reproducibility
• Known limitations
• Links
• Citation
• License

---

Motivation

Remote-sensing analysis is fragmented. Detecting change between two acquisitions, localising an object, captioning a scene, answering a question about it, and fusing optical with SAR each live in a different model, a different preprocessing convention, and a different output schema. Assembling them into one answer means re-solving the same problems — tiling, band handling, coordinate systems, confidence — every time.

SatQuery AI explores a single hypothesis: a small deterministic router plus a shared evidence contract can make a heterogeneous specialist ensemble behave like one system, without a large language model in the control path. The router understands the query. A deterministic policy decides which specialists run. The specialists compute. The evidence engine proves the answer.

Two design rules follow from that, and they are non-negotiable in the codebase:

• No LLM-generated coordinates. No LLM-generated confidence. Coordinates come from detection and segmentation heads; confidence comes from a calibrated scoring path.
• Every result carries an observable execution trace — never chain-of-thought.

Why not one end-to-end model

The design is a router plus specialists, not a single model that "does satellite QA". Each layer has exactly one verb, assigned in the architecture freeze:

router understands; policy engine decides; specialists compute; VLM explains; evidence engine proves.

That division is not decoration — it resolves real ambiguities about where a decision belongs. The Intent type makes the first row explicit in code:

class Intent(BaseModel):
    """Output of the learned router. Advisory only — the controller decides."""

The four constraints that force the modular design:

The cost of this design is that there is no system-level accuracy number — because there is no single model to measure. That absence is stated rather than papered over; see Known limitations item 1.

What the system is deliberately not

What the system supports

Six specialist tasks. All six are reported available: true by the live capability contract (GET /api/capabilities, probed 2026-09-25, schema_version 1.0).

The live contract also carries per-task notes — vqa/caption fetch SmolVLM weights from the Hub on first use, grounding fetches the RemoteCLIP encoder on first use, and optical_sar declares modalities: ["optical", "sar"].

The ontology is wider than the capability list

There are two related vocabularies, and they are not the same six:

• core/schemas.py::Task carries seven values: vqa, caption, grounding, change, optical_sar, change_vqa, unsupported.
• The router's label space (router/label_space.py::TASK_CLASSES) is six classes: vqa, caption, grounding, change, optical_sar, unsupported.
• GET /api/capabilities lists six tasks — the same six as the router's minus unsupported, plus change_vqa.

This asymmetry is intentional. unsupported is a routing outcome ("this is not a satellite-imagery question"), not a servable capability. change_vqa is reached through the change family rather than being a separate router class, and it is servable. The three sets are reconciled in one place — the frontend's ROUTER_TASK_TO_SERVER map (frontend/assets/js/mission.js) — because AnalysisRequest is extra="forbid" and any string outside the Task enum is a 422.

Supported inputs

Confirmed by the implementation, not assumed:

Modality is inferred server-side from band count, not from the file extension: {1, 2} bands ⇒ SAR, {3, 4, 8, 11, 12, 13} bands ⇒ optical. The browser cannot read band count, so the console warns when a submitted pair looks like two ordinary photographs rather than an optical/SAR pair.

Per-file upload limit: 4,194,304 bytes (4 MiB). Larger files are refused with HTTP 413 — imagery must be downscaled first.

The size cap is one number shared by two layers

The cap is not a literal in two places; both the gateway and the inference service read SATQUERY_MAX_FILE_BYTES, and the default is 4 * 1024 * 1024 in both. The inference service's _asset_max_file_bytes() (app/space_app.py) is deliberately strict about it:

• a missing variable ⇒ the 4 MiB default;
• a non-integer value ⇒ ValueError (not silently defaulted);
• a non-positive value ⇒ ValueError (a cap of 0 refuses every upload, which is a configuration error rather than a limit).

The reason is recorded in the source: a silent default would let a deployment whose operator typed a malformed cap keep accepting uploads against a limit nobody chose, while the gateway refused to boot for the same value — the two layers disagreeing about what "too large" means, which is exactly the failure the shared variable exists to prevent.

The accepted content types mirror the gateway's allowlist (defence in depth — the Space validates independently rather than trusting the gateway to be its only caller):

image/tiff, image/geotiff, image/png, image/jpeg, application/octet-stream

Uploads are handle-based, and the Space owns the bytes

POST /v1/assets accepts one file and returns an opaque ephemeral handle. The store lives on the inference host, not the gateway, because the inference host is where inspect_raster reads the bytes and where cache_max_models: 1 serialises their consumption — a gateway-side store would put the bytes on a different machine from the reader. The upload endpoint is off by default and enabled only when SATQUERY_ASSET_ENABLED and SATQUERY_ASSET_DIR are both set; otherwise it answers a named model_unavailable envelope explaining the switch. Handle capacity defaults to 32 and the handle TTL to 900.0 s, both overridable by environment (and read from the environment rather than configs/base.yaml on purpose — adding a key there would move the frozen config hash).

Input geometry and normalisation

Downstream of the format check, input handling is governed by the frozen registry (configs/base.yaml):

The tile policy follows the plan's section 9.1: whole-image thumbnail first, then top-K tiles. max_tiles is the hard ceiling on tiles examined; top_k_tiles is how many are actually dispatched — and the loader rejects a config where top_k_tiles > max_tiles.

Architecture

This is the actually deployed topology. An older direct-client-to-inference design is superseded.

flowchart TD
  B["Browser<br/>(static console)"] -->|HTTPS| CF["Cloudflare Pages<br/>satquery.pages.dev"]
  CF -->|"HTTPS JSON · /api/*"| R["Render<br/>satquery-orchestrator"]
  R -->|"outbound long-poll<br/>POST /tunnel/agent"| T{{"outbound tunnel"}}
  T --> C["GitHub Codespace<br/>FastAPI inference · CPU · :8000"]
  C --> S["Specialists"]
  S --> M["SmolVLM · RemoteCLIP · STANet-change<br/>CROMA-fusion · MiniLM router"]
  M --> E["Evidence engine<br/>+ temperature scaling"]
  E --> RE["ResultEnvelope"]
  RE -->|"tunnel → Render"| B

Why a tunnel

The inference host runs in a GitHub Codespace. The forwarded-port path is not reachable for a private repo (it returns HTTP 302), so the orchestrator keeps a long-poll tunnel: the Codespace dials out to POST /tunnel/agent and holds the connection; Render queues work onto it. transport_mode is auto, and the tunnel is the live transport. There is no SATQUERY_UPSTREAM_URL and no HF_TOKEN in the live configuration — the transport is the outbound tunnel, not a forwarded port.

The live health payload (GET /api/health, probed 2026-09-25) records the tunnel's state directly:

{"status":"ok","service":"satquery-orchestrator",
 "tunnel":{"agent_connected":true,"agent_id":"codespaces-fd1038","pending":0,"completed":97},
 "config":{"codespace_name":"potential-space-trout-r4ppw969w45j2pvvw\n","codespace_port":8000,
           "transport_mode":"auto","tunnel_timeout_s":150.0,"wake_timeout_s":120.0,
           "upstream_timeout_s":90.0,"device":"cpu","has_github_token":true}}

Note codespace_name still carries a trailing \n — that is item B-02, cosmetic, and still open.

The nine-state controller

The inference host is a FastAPI service built by build_space_app(). Behind the transport sits a deterministic controller with a nine-state finite state machine (core/schemas.py::ControllerState, mirrored in configs/base.yaml §agent.states):

RECEIVE → PARSE → VALIDATE → PLAN → PREPROCESS → EXECUTE → AGGREGATE → VERIFY → RESPOND

The controller is the only component that dispatches. agent.max_specialists is 4, agent.timeout_seconds is 120, and agent.unload_after_workflow is true — the controller unloads models after a workflow so that cache_max_models: 1 is honoured rather than thrashing the cache.

Frozen backbones, trained modules

Four backbones are pinned by repo_id + revision and fetched from the Hub on first use. Nothing is fine-tuned end-to-end.

Two consequences follow: the system is small (the six trained artifacts total ~125 MiB; everything else is public weights), and no backbone weights are redistributed by this release.

The evidence and confidence stages

Specialists each emit Evidence for what they computed. The EvidenceEngine (evidence/engine.py) does not re-derive any of it; it aggregates:

collect across specialists → deduplicate → order → renumber → cap

The pipeline is pure and deterministic — no clock, no RNG, no I/O — and its purity is a test assertion, not a hope. Three details matter:

• Stable identity. Evidence.evidence_id defaults to a random uuid, which is useless across runs. The engine renumbers to evidence_001, evidence_002, … (zero-padded to three digits, well past the evidence.max_items bound of 32) so a downstream artefact can cite one item deterministically.
• Defined order. Items are sorted by (type, source_specialist, score DESC, coordinates). The coordinate tie-breaker is what makes the order total; without it, two items sharing type, specialist and score would fall back to Python's stable-sort insertion order, reintroducing input-order dependence.
• Content identity, not container identity. Dedup keys on (type, source_specialist, coordinate_system, rounded coordinates, rounded score) — payload, artifact_ref and evidence_id are deliberately excluded. Two items that agree on the same geolocation, one carrying a crs and one not, have made the same claim about the world; the surviving item's payload is merged with the discarded one's so the crs is not lost. Two items that share a type and score but disagree on coordinates are two different claims and are both kept — suppressing a spatial disagreement would be the silent contradiction the freeze forbids.

The engine records dropped_duplicates (non-zero is normal and healthy — it means two specialists agreed), dropped_over_limit (non-zero is a warning — a specialist's evidence did not survive), and truncated. evidence_digest() exists so that reproducibility is an assertion:

aggregate(inputs_a) is reproducible iff digest(a) == digest(b)

The confidence stage is evidence/confidence.py, and it exists to enforce one rule: a calibration that claims to be fitted when it is not is a false claim of reliability. So when no fitted artifact is available it passes the raw score through unchanged, sets method="uncalibrated", and leaves calibrated=None — which is what makes it honest, because ConfidenceBreakdown.value then returns raw, and any consumer can distinguish "we calibrated this" from "we did not". The temperature scaling itself is calibrated = sigmoid(logit(z) / T), with two stated failure modes handled explicitly: a fitted T of exactly 1.0 is the identity map and is reported as uncalibrated rather than silently pretending to have done something, and z = 0.0 (whose log-odds diverge) is clamped at the boundary so a hard zero cannot become a NaN.

Repository map

Scope of this repository. It contains the documentation and the source code only. It deliberately does not contain: datasets, the artifacts/ tree (trained weights, checkpoints, evaluation outputs), Jupyter notebooks, the static frontend, logs, or any credential. The six trained artifacts are published on the Hugging Face Hub; the frozen backbones are fetched from the Hub at run time. Some documents cite the source repository's internal phase reports by filename as provenance — those files are not part of this release.

Not in this repository: artifacts/, data/, notebooks/, frontend/, logs/, reports/.

The component inventory in full (every path is the authoritative location):

The four endpoints

The inference service serves exactly four JSON endpoints (app/space_app.py). No Gradio GUI exists in code; the file serves JSON only, by design, so it cannot compete with the static frontend's contract.

The gateway in front of it exposes the same functionality under /api/* and holds the request-side security boundary. Two details are worth recording because they cost real debugging time:

• Framework-raised errors carry the same envelope. An unmatched route (404) and a method mismatch (405) are wrapped by a Starlette exception handler so they answer with the contract's error shape (routing_error, recoverable: false) rather than FastAPI's default {"detail": …}. An unhandled exception answers with satquery_error and a fixed, operator-safe message; the exception's own text is logged server-side only, so a traceback cannot disclose internal paths to an unauthenticated caller.
• The upload body is read bounded. POST /v1/assets uses read_body_bounded(request, cap) rather than await request.body(), so the cap is applied while reading rather than after the whole body has been buffered. The measured defect this fixed: with the cap at 1 MiB, a 64 MiB body produced a peak allocation of 128 MiB, tracking body size linearly with no ceiling, and the 413 came only after everything had been held.

Documentation map

The architecture reference is a hub plus ten deep sub-documents. Every one of them is written at long-form depth, with real signatures, schemas, numbers and file paths.

Companion references, all in this repository:

Routing and the execution trace

Routing is deliberately two-stage, and the split matters:

1. interpret() — the reading. A lexical pass over the query produces a reading: task intent, modality, temporal requirement, spatial scope, and expected evidence kind. It is asset-count-blind.
2. chooseTask() — the dispatch. The reading is combined with the number of attached assets to decide the task actually dispatched. This is why a reading of change with one asset dispatches change_vqa — the documented quantifier upgrade.

The server-side router has the same two-part shape in Python: router/classifier.py::IntentRouter.route() produces an Intent, and core/planner.py turns that Intent into an ordered ExecutionPlan. The planner's docstring states the rule it exists to enforce: the planner is the only component permitted to choose what runs; its input is a router prediction, its output is a plan, and the router's Intent is an input to a decision, never the decision. A design where intent.task selects a specialist in one step has collapsed understand into decide, and three things break: the router can never be overruled, a query needing two specialists can never get both, and there is no auditable record of the decision.

The five router heads

The learned router is a small adapter over frozen MiniLM embeddings. router/label_space.py is the single source of truth for its output ontology — both the dataset generator and the training script import from it, so a class added in one place cannot silently desynchronise the other.

The three binary heads use a single logit rather than a two-way softmax: a 2-way softmax would waste a parameter and make the loss harder to weight. Because the heads are independent by construction, a confident task label with an incoherent binary head is possible; the classifier resolves that in favour of the task label (the controller keys off the task), and records that it did so.

Router configuration (configs/base.yaml §router):

Finding F4-1 — the tokenizer ceiling. The MiniLM tokenizer's own ceiling is 256 (verified by probe). The project truncates to 128 — a deliberate truncation well inside the ceiling, not the model limit. Satellite queries are short; halving the sequence halves attention cost for no measurable accuracy loss. The encoder asserts max_length ≤ 256, because truncating above the ceiling is a silent no-op.

Finding F4-2 — the router needs no GPU. The encoder is frozen, so embeddings are cached and the 50,822-parameter adapter trains on cached vectors. Measured on CPU: 20 epochs / 4,096 vectors in 0.28 s.

Finding F4-3 — splits must be by group. Splits are by group (template / hard-negative family), never by example. Hard-negative families are placed in the test split so their accuracy measures generalisation rather than memorisation; splitting by example would leak template variants across the boundary.

The confidence gate and the fallback

IntentRouter.route() runs the learned router first, and falls back to the lexical rules only when the learned router is below the confidence gate (or when the encoder cannot be loaded at all):

query
  → encoder.encode          (frozen MiniLM, 384-d)
  → adapter                 (5 heads, softmax / sigmoid)
  → confidence gate         (router.confidence_threshold = 0.70)
  → lexical fallback        (only if below the gate)
  → Intent                  (validated pydantic model)

The router never refuses to answer; above_threshold carries the uncertainty, and the planner — not the router — decides what to do about it. The fallback is purely lexical: no model, no embeddings, no randomness, ordered rules with the highest specificity first, and it never invents capability — if nothing matches it returns unsupported with low confidence rather than guessing a task. Its precedence is explicit, and it matters because the phrasings overlap:

dual_modality > temporal > spatial > caption > vqa > unsupported

Two examples of why precedence is load-bearing, both from the source:

• "show me where the change happened" has both a spatial term and a temporal term → change + spatial_output=True.
• "compare optical and radar to locate built-up areas" has dual-modality and spatial → optical_sar (spatial does not apply to the joint workflow, whose output is a classification).

The fallback's confidence band (0.72–0.92) overlaps and can exceed the trained model's — on the spec's own examples the fallback returns 0.850–0.920 against the trained model's 0.780–1.000. So the planner applies a provenance discount to its own reading of the confidence and never edits Intent.confidence itself: a lexical fallback at 0.9 is not the same evidence as a learned model at 0.9, and treating them identically would let a matched keyword outrank the model it fell back from. IntentRouter.adapter_source returns 'trained' or 'lexical_fallback' so a caller can check this in one place instead of inferring it from a confidence band — because from_config defaults adapter_path to None, meaning the default router runs the fallback, not the trained adapter.

A router bug worth recording

An earlier revision evaluated the temporal rule before the location rule, so "Where are the built-up areas in this image?" matched \bbuilt\b as a change marker and area inside "areas" as a quantifier. With one asset it collapsed to vqa and answered "River". Fixed on 2026-09-25 in frontend/assets/js/mission.js; the fix is covered by regression tests and verified live. The same defect existed on a second surface (SQ.policy in core.js) and was fixed the same day.

The fix is four lexical changes, each documented in the source because each was a real failure:

Both defect queries now dispatch to grounding and are captured in the screenshot set below.

The eight execution events

The console renders an execution trace built from eight events, emitted by the frontend around real network calls (SQ.EVENT_NAMES in frontend/assets/js/core.js):

These are a frontend vocabulary driven by observable events — not a backend protocol and not a model's reasoning trace. The run engine is deliberately dumb: it renders whatever events it receives, and swapping the mock driver for a websocket/SSE feed of the same event names is the entire integration surface. On live runs the trace bar reaches 94.4444 % (17/18) and every node is marked live; the preview path is the only source of mock-marked nodes.

The trace is deliberately not chain-of-thought. ExecutionTrace (core/schemas.py) carries run_id, schema_version, task, query, inputs, modalities, intent, validation, workflow, steps, selected_models, parameters, outputs, confidence, timings, fallbacks, errors, contradiction, config_hash, started_at, finished_at — observable facts, with no field for model reasoning and no LLM-generated confidence. The router's own trace projection is the model of this: it emits the task, modality, the three booleans, the rounded confidence, the source, above_threshold, used_fallback and fallback_rule — and nothing else.

Real inference vs. the preview path

• Real path (production). With assets attached, the console calls POST /api/infer on the Render orchestrator. Every run returns a real run_* identifier from the inference service. Live validation recorded 0 mock nodes across 24 live runs.
• Preview path. With no files selected, the console renders a labelled illustrative preview so the interface is explorable without the stack awake. Preview nodes are explicitly marked is-mock and never appear in a live run.

The distinction is observable, not asserted: a live run shows live · N evidence · transport … and zero .trace__node.is-mock elements.

Two related traps are worth stating because they are the kind of thing a reader will otherwise misdiagnose:

• A pair-requiring task with one asset is refused, not silently downgraded server-side. With one asset, change answers invalid_request ("change requires exactly 2 assets (T1 and T2); got 1") and the whole envelope comes back degraded: true. Measured live, 2026-09-25. The console's job is to avoid asking for a pair-requiring task when only one asset exists; it does so in chooseTask(), and it names the substitution rather than hiding it.
• The asset set sent is per-task. vqa, grounding and caption accept a single image, while change, change_vqa and optical_sar require a pair. Sending the optional earlier frame to a single-image task makes the backend reject the whole request — measured live when a pair was uploaded and a VQA question asked. The fix isolates the file set so the pair is only ever sent to the tasks that declared it.

Models

Six trained artifacts are released. Four are task heads and two are adapters — none is a complete standalone model, and each documents its backbone dependency. Full detail: docs/MODELS.md, MODEL_CARD.md, and the generated models/manifest.json.

Backbones are third-party and pinned by repo_id + revision in configs/base.yaml; they are fetched from the Hugging Face Hub, not redistributed here.

The six artifacts, byte-for-byte

models/manifest.json is generated by reading the files — no byte count or hash is typed by hand. Its schema is satquery_model_manifest_v1, generated 2026-09-25T18:15:38+00:00, artifact_count: 6, and every entry carries the frozen config_hash 78f1e3700da15aa1.

Total released weight payload: 131,130,325 bytes (~125 MiB). Two of the six hashes are cross-checked against values recorded independently elsewhere in the project — change_vqa_head against artifacts/change_vqa/run/PROMOTION.json, and vlm_lora_adapter against the adapter's own provenance manifest — and both agree. That is an external cross-check, not a self-consistency claim.

The manifest also records each artifact's architecture and the metric artifact it came from:

Training checkpoints also exist (checkpoint_last.pt at 189,291,829 B for change; checkpoint_last.pt at 12,640,331 B for grounding; checkpoint-1500 / checkpoint-2000 for the LoRA adapter) and are not the released artifacts — they are archived as provenance.

The Hugging Face release

The six artifacts are published at https://huggingface.co/thundercode/SatQuery (public, private: false, gated: false), HEAD 55681e0cddb91a4a5655da98a49bc025e537b657, 22 files, last modified 2026-09-25T18:21:52Z.

The release was verified by re-downloading each artifact over direct HTTPS and hashing the bytes received, rather than trusting the upload step: 6/6 MATCH, 0 failed, and the four support files (README.md, MODEL_CARD.md, models/manifest.json, models/checksums.sha256) confirmed present. The pre-existing content was a 25-byte stub README (literally ---\nlicense: unknown\n---) which was replaced, and the standard HF LFS routing .gitattributes, which was left untouched.

A verification method that was itself wrong (recorded). The first verification attempt reported all six artifacts DIFFER, with every remote hash equal to e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855 — the sha256 of empty content. The cause was not the upload: hf_hub_download returned an empty file in this environment, so the verifier hashed nothing. It was caught by a second, independent method (a direct curl download), which produced the correct hash for router/adapter.pt and confirmed it to be a real PyTorch zip (PK\x03\x04). The verifier was then rewritten to use direct HTTPS with proxies disabled. The failed first attempt is recorded because a verifier that silently hashes an empty file would have produced a false failure — and, with a different bug, could just as easily have produced a false pass.

No secret was uploaded. The uploaded set is the model card, the manifest, the checksums, the docs, and the six weight files; no tokens, keys, environment files or credentials exist in any uploaded file, and the token used for the upload is not written into any released file.

Measured results

Every number below traces to an artifact, a test, or a live run. Nothing here is a system-level benchmark — no such benchmark exists (see Known limitations).

Source artifacts and the rules the table follows

Each metric family has exactly one source artifact:

All 20 quoted metrics are checked against these files by release/tools/verify_readme_metrics.py, which resolves nested artifact keys (including keys that themselves contain dots — the recall dict is keyed "0.10"/"0.25"/"0.50", so a naive split(".") walk would break) and compares each value at the precision printed here. It exits non-zero if any claim fails and prints ALL CLAIMS VERIFIED only when everything matches. The table obeys six rules:

1. Two protocols are never collapsed. Grounding is reported under both the canonical and matched6 protocols. Quoting 0.2838 alone would be selective.
2. Two test sets are never collapsed. Change-VQA is reported on test and test2.
3. accuracy never travels without macro-F1. For imbalanced multi-class heads (optical-SAR, change-VQA) the macro-F1 is reported alongside accuracy, always.
4. Validation is not test. The router number is labelled "overall ungated accuracy", val, n = 86.
5. A negative result stays negative. Calibration ECE worsened and is shown worsening.
6. USABLE ≠ ACCEPTED. The VLM metrics are real; the artifact is nevertheless acceptance-rejected.

There is also no composite or vanity score: no single headline accuracy for the system, and none invented by averaging the per-task numbers.

Grounding: three decode variants, two protocols

The grounding head is evaluated under two matching protocols (canonical, matched6) and three decode variants. Quoting a single number would misrepresent the result, so all of them are listed:

The head clears the zero-shot baseline, but only the threshold decode is meaningfully above it — the argmax decode (0.1215) is barely better than zero-shot. The absolute level is modest either way: grounding is useful, not solved.

The [email protected] numbers travel with the IoU numbers: canonical 0.2198, matched6 0.1938. The box convention is a common source of silent error, which is why the project converts VRSBench's 0–100 boxes to its own 0–1 convention through a declared benchmark_box_scale: 100.0 — so the conversion cannot be applied twice or forgotten — and reports both protocols.

The head itself is a trainable head over the frozen RemoteCLIP ViT-B/32 encoder. Its per-cell feature is concat([patch, text, patch·text, global_pool]) = 4 × 512 = 2048 (finding P7-1: the transformer width is 768, but visual.proj maps to a projected dim of 512). Cells are assigned by ground-truth box centre (cell_relative decode). The objectness BCE is weighted 20× because only ~1 of 49 cells is positive; unweighted, the optimum collapses to "no object" everywhere. Image resolution is frozen at 224 — 448 was evaluated and rejected (see below).

The grounding resolution decision — a pre-registered rejection

Question: should grounding decode at 448 or 224? Answer: 224. 448 was rejected — and the rejection is notable because it was pre-registered and then confirmed by a paired test over identical samples (n = 16,159):

448 lost on every axis. The pre-registered decision rule and the paired test agree on 224. This is a model of how a resolution decision should be made: declared in advance, then tested. Recorded as RESOLVED 2026-09-16 in configs/base.yaml and in docs/RESEARCH_NOTES.md §2.

Fusion: measured but the ruling is open

Optical-SAR fusion reaches 0.931 accuracy on a 19-class held-out set of 4,000 — but only 0.434 macro_F1. Those two numbers describe very different things: the model is accurate on frequent classes and weak on rare ones. The acceptance ruling for this head is OPEN, and the headline accuracy must never be quoted without the macro_F1 beside it.

The metric JSON records why the macro score is low by construction: of the 19 classes in the label space, 14 are present and 5 are absent in the scored split, and the macro-F1 denominator is all 19 (absent classes contribute 0.0). It also records is_deciding_statistic: False — this is a reported measurement, not a decision statistic. The feature concatenation is the frozen one:

input_dim = 3 × 768 + 12 + 2 = 2318   →   hidden 512   →   num_classes 19   (BigEarthNet CLC)

and the availability mask is consumed by the fusion head, not by CROMA (finding C-1) — CROMA always sees the canonical channel counts (12 optical, 2 SAR).

Two further caveats on this head, stated rather than hidden:

• The live service returns a bare class index (class_18), not a human-readable label. The modality accounting in the response confirms the right channels reached the fusion head, but the presentation is not user-facing.
• The local BigEarthNet subset is 100 % single-label, against the official 1–11 multi-label scheme, so its metrics are not comparable to published BigEarthNet numbers. Any statement of the form "BigEarthNet mAP = X" is false for this subset.

Change-VQA: two test sets, and they disagree

artifacts/change_vqa/run/PROMOTION.json records two test evaluations:

The test numbers are the higher pair. Both are reported here; quoting only test would overstate the result. The acceptance ruling is OPEN.

The head was trained outside this repository, on an external GPU (Kaggle), and promoted through a byte-identity gate: sha256 cfae5e43b97ca930f568dc5b8ae4f36b24e9ff717af226159802206ffd63a82a, 5,822,809 bytes, architecture change_vqa_head_v1, 1,453,912 parameters, 0 non-finite tensors, weights not modified during promotion, byte-identical to source, hash agreeing across model_metadata.json, run_record.json and hashes.json. Selection was epoch 8, chosen on val answer accuracy 0.700018, stopped by early stopping; seed 42. It trains on cached change + text features, not on raw imagery — the raw CDVQA loader loads examples but has no training loop of its own, and the two paths are not conflated.

Phase 6 / VLM: deployment success ≠ model acceptance

The Phase-6 SmolVLM LoRA adapter reaches exact_match 0.963 and F1 0.96432 on a frozen 1,000-question subset. It is marked USABLE_VERIFIED and ACCEPTANCE-REJECTED.

Those two verdicts are not in conflict, and the distinction is the point:

• USABLE_VERIFIED — the adapter loads, runs, and produces the measured numbers in the deployed pipeline. The F1 is +49.5 pp over the unadapted baseline.
• ACCEPTANCE-REJECTED — the change did not clear the project's own pre-registered acceptance bar. The record of why is stored in artifacts/vlm/phase6_closure.json under why_acceptance_rejected; the artifact's status is CLOSED.

A model can be a working engineering artifact and a rejected research result at the same time. This release keeps both labels. The deployed caption/VQA path therefore uses the unadapted SmolVLM.

The adapter's own shape is recorded: PEFT 0.19.1, r = 16, alpha = 32, dropout = 0.05, targeting model.text_model.*.{q,k,v,o,gate,up,down}_proj, precision fp16 (finding C-6: the T4 is compute capability 7.5, so fp16 — not bf16), batch size 2, gradient accumulation 8, learning rate 0.0002, 1 epoch, gradient checkpointing on, save_every_steps 500.

Calibration: it got worse, and we say so

The change_vqa confidence path applies temperature scaling (T = 0.9773). Measured on the validation split (n=16441):

Calibration did not improve — it moved slightly worse. The fitted temperature is 0.9772732 and ece_improvement is −0.001174: negative. The scaling is retained because it is part of the frozen configuration, not because it helped. The reliability curve plotted on the Benchmark page is explicitly labelled as the pre-scaling diagram so a reader cannot mistake it for the calibrated result. The calibrated curve is not plotted.

What is NOT benchmarked

Live validation

Validation drove the production site in a headed browser, one upload per case, with per-case screenshots and recorded run identifiers. It is behavioural evidence — that the pipeline runs and routes correctly — and it is not an accuracy claim; accuracy and behaviour are evaluated separately.

Each pass produced fresh run identifiers — no run id is shared between passes. The three passes ran against two frontend revisions:

The harness asserts the form state before dispatch — that the query box really holds the intended query, that #obsTail reads ready, and that both frames are attached for pair tasks. This matters: an earlier harness revision typed with synthetic key events that Chrome silently drops when the window lacks OS focus, so it dispatched the page's default query and still recorded a "result". The assertions exist because of that failure. The earlier 8/8 run was independently re-examined and confirmed not to have been infected (its answers were query-specific and the query text was embedded in the answers), but the failure mode is recorded because it is exactly the kind of silent false-positive an evaluation harness must never have.

Deployed-artifact integrity was checked separately: 9 files were re-read from the GitHub API and compared byte-for-byte against local copies, and all 9 were sha256 byte-identical; the deployed HEAD was re-read from the API.

Representative real run IDs

One full pass (pass 3 of 3) — the same pass the screenshots below are drawn from. Run identifiers are fresh on every pass; the other two passes recorded different ids.

The last two are the router-defect queries. Both previously collapsed to vqa and answered "River".

Note the fifth row: "Did the coastline advance between the two observations?" is read as change and dispatches change_vqa — the documented quantifier upgrade, because the page's Answer block promises an answer and the server's change returns a spatial map with no language output. With one asset attached, change would instead be refused outright ("requires exactly 2 assets"); the console avoids asking for a pair-requiring task when only one asset exists.

Screenshots

Eight captures from the post-fix live run (headed browser, 1384×855, one upload per case). Each panel shows the run identifier, the frozen config hash 78f1e3700da15aa1, and the evidence list returned by the specialist — nothing is mocked.

All eight are reproduced byte-for-byte in the evidence archive (Phase 7) with SHA-256 recorded in RELEASE_MANIFEST.md.

Installation

Python 3.11+ and a CPU are sufficient. No CUDA requirement — device is selected via SATQUERY_DEVICE; all placement is .to(device), never .cuda().

git clone https://github.com/Anish-lab-blip/SatQuery-AI
cd SatQuery-AI
python -m venv .venv
source .venv/Scripts/activate      # Windows git-bash; use .venv/bin/activate on Linux/macOS
pip install -r requirements.txt

Backbones are fetched from the Hugging Face Hub on first use, pinned by revision in configs/base.yaml. The frozen config hash is 78f1e3700da15aa1 — the loader refuses to run a config that violates the recorded invariants (for example fusion.input_dim == 3*encoder_dim + 12 + 2).

The invariants the loader enforces

core/config.py validates the registry at load time and raises ConfigError — naming every violation — rather than letting a bad value reach runtime. The invariants are not documentation; they are checks:

Because a config edit moves Config.hash and invalidates every artifact keyed to it, deployment state that must not move the hash (asset-store capacity, TTL, the per-file cap) is read from the environment rather than from configs/base.yaml — the same reasoning that keeps the config hash frozen.

Local development

# Inference service, CPU (this is the launcher the Codespace runs)
PORT=8000 python deploy/codespace/serve.py

# Health
curl localhost:8000/v1/health

The frontend is fully static and needs no build step to serve locally:

python -m http.server 5500 --directory frontend

Run the frontend regression suite:

python -m pytest tests/unit/test_frontend_live_wiring.py -q

The test suites

The full-suite failures are not hidden, and they are not regressions: 4 are the sandbox's bulk-delete guard (test_safe_delete_shim), 1 is an ordering flake that passes in isolation, and 1 is a stale adapter test (CROMA is now shipped). Re-running the affected files together gives 137 passed, confirming the failures are attributable to the sandbox environment and test ordering rather than the code under test.

Reproduce a live run

The deployed stack is reachable. Note the authoring sandbox has a dead proxy, so outbound calls need --noproxy '*' (curl) or ProxyHandler({}) (Python):

curl --noproxy '*' https://<backend-host>/api/health
curl --noproxy '*' https://<backend-host>/api/capabilities

/api/capabilities returns six tasks, all available: true. A live run requires the tunnel agent to be connected (agent_connected: true); if the Codespace is stopped, the request parks until the tunnel timeout. See docs/DEPLOYMENT.md §5–6.

Deployment

The live topology is Cloudflare Pages → Render → outbound tunnel → GitHub Codespace.

Deployment sources are separate repositories from this release. The wake flow is: Cloudflare → Render → start the Codespace if stopped → poll /v1/health → surface "Waking inference engine…" → POST /infer → result.

Live revisions at this release

Trap. deploy/ inside the monorepo is stale and untracked. It is not the deployed source. Edits must go to the three real repositories. Recorded in docs/DEPLOYMENT.md §1.

Environment variables

Render (gateway), measured live:

Codespace (inference):

Deploy mechanics: the frontend is staged by scripts/stage_pages.mjs and deployed with npx wrangler pages deploy; the backend is a render.yaml blueprint whose main.py exposes app; the inference host runs deploy/codespace/serve.py on $PORT and the devcontainer forwards port 8000 and starts the tunnel agent on postStartCommand. Repository writes are performed through the GitHub Git Data API (blob → tree → commit → PATCH ref) with sha256 byte-verification of every uploaded blob, rather than git push, so each deployed file is verified by content hash.

Deployment caveats

• Cold start. The inference host may be stopped when idle. The first request after a cold start can exceed the client timeout while weights are fetched; a retry a few seconds later normally succeeds. Warm the stack before any demonstration and confirm GET /api/health reports tunnel.agent_connected: true. Cold start is tens of seconds and is documented rather than papered over.
• Tunnel gaps. The tunnel agent can be briefly absent. A request issued during such a gap may hang or return HTTP 504. This is not fixed in production — a prepared patch (forward_unavailable 503 / upstream_timeout 504 plus a codespace_name fix) exists and is documented, but it was deliberately not deployed. Root cause: in auto transport mode a tunnel timeout falls through to the forwarded-port path (main.py:546), which then spends the 120 s wake timeout on an HTTP 302 — the observed ~249 s failure (150 + 120).
• codespace_name is still reported with a trailing newline by /api/health (cosmetic; the wake path strips it).
• Never retry POST /api/infer at the gateway — a retry consumes inference twice.
• Platform traps, recorded so they are not rediscovered. Cloudflare _headers rules concatenate rather than override, and Chromium takes the first max-age it encounters, so a later rule cannot "fix" an earlier one. Cloudflare 308-redirects X.html → /X, so the extensionless path must be referenced. A forwarded Codespace port returns 302 for a private repo — which is why the tunnel exists. And the tunnel agent must be started by the devcontainer's postStartCommand, or a restarted Codespace comes up with agent_connected: false.

Historical context

The superseded design ran inference on an HF Space with ZeroGPU behind a Railway gateway. The active design moves to Render + Codespace, CPU-first, with an outbound tunnel. The four-endpoint contract, the gateway responsibility table, the env-var vocabulary and the config freeze are unchanged — only host names moved. configs/deploy.yaml still describes the old HF-Space/ZeroGPU target and is left undisturbed as frozen paperwork (editing it would move the config hash); no Gradio runtime exists in code. The declared ZeroGPU durations are transcribed, not invented — app/space_app.py carries GPU_DURATIONS = vqa 20, caption 20, grounding 45, change 30, optical_sar 45, change_vqa 30, and a task with no declared duration is a programming error rather than a default, because silently picking one would reserve the wrong amount of the 5 GPU-minute daily budget. The decoration has never executed here (spaces is not installed in this environment); configs/deploy.yaml sets cpu_mode_required: true, so a CPU run must work, and it does.

Reproducibility

1. Configuration. configs/base.yaml is the single registry; no magic numbers in Python. Its hash is recorded in every execution trace. Frozen hash: 78f1e3700da15aa1. A config edit moves the hash and invalidates every artifact keyed to it.
2. Backbones. Pinned by repo_id + revision, never by floating tag: HuggingFaceTB/SmolVLM-500M-Instruct@a7da5b986cb5, chendelong/RemoteCLIP@bf1d8a3ccf2d, sentence-transformers/all-MiniLM-L6-v2@1110a243fdf4, antofuller/CROMA@0dd28e3d633b.
3. Released artifacts. models/manifest.json and models/checksums.sha256 are generated from the actual files — never hand-typed. Verify with sha256sum -c models/checksums.sha256.
4. Splits. LEVIR-CD-256: train 7120 / val 1024 / test 2048. Grounding: VRSBench n=16159. Fusion: held-out test n=4000. Change-VQA: test n=39686. Leakage isolation is by scene_id.
5. Prompts are versioned files, frozen before benchmark evaluation.
6. Negative results are preserved. Rejected and open rulings are recorded, not removed.

The frozen contract, in the registry's own vocabulary:

Reproduce the metric check (cheap, no GPU)

python release/tools/verify_readme_metrics.py

It reads the artifacts under artifacts/, compares each of the 20 quoted metrics at the precision printed in this README, and also asserts the statuses (that the VLM headline contains ACCEPTANCE-REJECTED; the router's corpus_limited / n_val; the calibration temperature and ece_improvement). It exits 0 and prints ALL CLAIMS VERIFIED only when everything matches.

What "reproduce" means, and what it does not

For the two externally-trained artifacts, the repository reproduces the promotion gate (byte-identity, sha256, zero non-finite tensors), the evaluation, and the serving wiring; it does not ship a one-command retrain. That is stated rather than implied.

What is NOT reproducible from this release

Environment traps worth recording for anyone reproducing: the authoring sandbox has a dead proxy (outbound calls need --noproxy '*' for curl or ProxyHandler({}) for Python); pytest is installed only in the repository virtualenv (.venv/Scripts/python.exe); the full-suite run trips the sandbox's bulk-delete guard; Cloudflare 308-redirects X.html → /X; and Chrome drops synthetic CDP key events when the window lacks OS focus, which is relevant to any browser-driven reproduction of the live validation.

Known limitations

1. No system-level end-to-end benchmark exists. Per-specialist metrics are real; a single end-to-end number is NOT RUN.
2. Router accuracy is validation-only (n=86, corpus-limited). Its test set was never run.
3. Optical-SAR fusion returns a bare class index (class_18), not a human-readable label. The modality accounting in the response confirms the right channels reached the fusion head, but the presentation is not user-facing.
4. VQA is weak-but-related. Asked what terrain dominates a scene, it answers "Grassland".
5. Fusion macro_F1 is low (0.434161) against 0.931 accuracy — rare classes are poorly handled. Of the 19 classes, 5 are absent from the scored split and contribute 0.0 to macro-F1 by construction.
6. Grounding IoU is modest (0.2838 canonical, 0.2566 matched6) — useful, not solved — and it is protocol-sensitive: the argmax decode (0.1215) is barely above the zero-shot baseline (0.0972).
7. Calibration makes ECE slightly worse (0.013755 → 0.014929), and is retained only because it is part of the frozen configuration. The calibrated reliability curve is not plotted.
8. Router lexical residuals. "What is the new runway?" reads change rather than vqa (the new-as-change heuristic fires outside where questions), and "How much built-up area was added?" reads vqa (under-trigger). A lexical router cannot cleanly separate "the new X" from "what's new"; a trained intent router exists in artifacts/router/ but is not attached.
9. B-07 tunnel gaps are not fixed in production (see Deployment caveats).
10. No license has been selected for this repository. Until one is, the artifacts carry license: unknown and no reuse rights should be assumed. This is an open owner decision.
11. The Anatomy of a Run page renders a recorded run whose plate uses the 720×720 variant of an image analysed at 730×730 — identical content, scaled by the canvas, but the "actual analysed image" wording is slightly loose.
12. The BigEarthNet local subset is single-label (100 %) against the official 1–11 multi-label scheme, so its metrics are not comparable to published numbers.
13. The VLM adapter is not accepted — metrics usable (exact_match 0.963), status acceptance-rejected; the deployed path uses the unadapted model.
14. The deployment repos are private, so their links 404 for an outside audience — by design.

Explicit non-claims

• No claim of state-of-the-art performance on any benchmark.
• No claim of production readiness for model quality — the deployment runs, but the models carry the limitations above.
• No claim that the trained heads generalise beyond their training-family test splits.
• No claim that calibration improves confidence.
• No claim that the VLM adapter is accepted for production use.
• No system-level accuracy is claimed anywhere, and none is produced by averaging the per-task numbers.

Links

Third-party models this work builds on (pinned, not redistributed):

Datasets referenced by the evaluations: LEVIR-CD-256 (change), VRSBench (grounding), BigEarthNet (optical-SAR fusion, 19 CLC classes), CDVQA + SECOND (change-VQA). No dataset is redistributed here.

Documentation in this repository:

Supporting and subsystem documentation:

Honesty rule. Every document in this repository uses one status vocabulary — IMPLEMENTED · VERIFIED · MEASURED · ATTEMPTED · NOT RUN · BLOCKED · DEFERRED · REJECTED · OPEN · RESOLVED · CLOSED — and negative results are recorded rather than omitted. Where evidence was missing, the text says UNKNOWN — not established from the available evidence instead of guessing.

Citation

No paper accompanies this release. Until one exists, cite the repository:

@misc{satqueryai2026,
  title  = {SatQuery AI: an interactive vision-language assistant for
            multimodal remote-sensing image analysis},
  author = {SatQuery AI contributors},
  year   = {2026},
  url    = {https://github.com/Anish-lab-blip/SatQuery-AI}
}

License

Not yet selected. See limitation 10. Backbone models remain under their own upstream licenses.

No LICENSE file exists in this repository. Until one is selected, the released artifacts carry license: unknown and no reuse rights should be assumed. This is an open owner decision, recorded as OPEN in docs/LIMITATIONS.md §5 and docs/CHANGELOG.md. The six trained artifacts are small modules over frozen backbones; the backbones are not redistributed here and remain under their own upstream licences — consult each backbone's Hugging Face page.