Why Far Looks Up: Probing Spatial Representation in Vision-Language Models

A lightweight framework for diagnosing how vision-language models internally represent spatial relations — left / right, above / below, far / close — beyond what benchmark accuracy reveals.

It implements the contrastive probing methodology from our paper "Why Far Looks Up: Probing Spatial Representation in Vision-Language Models": by swapping the two objects inside a spatial question and measuring the difference between the resulting hidden states (Δ vectors), we obtain a representation-level view of how cleanly a model encodes a spatial axis, and to what extent vertical and distance directions are coupled (the perspective heuristic "higher in the image ⇒ farther away").

TL;DR. Two text-only forwards per image (one question + its swapped version), one forward hook on every transformer layer, and a handful of numpy operations on the resulting Δ vectors give you axis-coherence curves, 6×6 delta-similarity heatmaps, 2D/3D PCA plots, and a complementary VD-Entanglement Index for any VLM you can load with 🤗 transformers.

---

What this code does

For every spatial question in EmbSpatial-Bench, the script builds a minimal contrastive pair by swapping obj1 ↔ obj2:

Original: "Is the [cup] to the left or right of the [book]?"   → left
Swapped : "Is the [book] to the left or right of the [cup]?"   → right

It then runs both questions through the VLM, captures the last-token hidden state of every transformer layer, and forms the per-sample delta:

$$\Delta r = h(\text{swapped}) - h(\text{original})$$

These Δ vectors are the basic unit of analysis. From them the script computes:

Everything is saved as .npz / .csv / .json so you can re-plot or run new analyses without re-running inference.

---

Release

Exact HF repo IDs for each scale are registered in probing.py (MODEL_REGISTRY).

---

Supported models

The framework ships with extractor wrappers for the following VLM families. Adding a new family is two short methods + one registry entry (see Adding a new model).

Layer selection

Paper-reported metrics use a single layer L* per model — picked (paper App. D.3) by three textual criteria: (i) axis coherence on all three axes is at or near its plateau, (ii) VD-EI is on a stable plateau rather than in a transient region, (iii) avoid the last few layers, which specialise on next-token prediction rather than rich representations.

For the four registered families below, L* is already stored on MODEL_REGISTRY[<model_type>] as paper_layer / paper_plateau / total_layers, so you can skip straight to inference. To pick L* for a new model:

1. Run probing once.
2. Open plots/metrics/axis_coherences.png and plots/metrics/vd-ei.png and read off the layers that satisfy the three criteria above.
3. Per-layer 3D PCA panels under plots/pca_3d/ are also saved for reference if you want additional visual context.

For a programmatic short-list, python probing.py --model_type <m> --recommend-layer reports a candidate range and a top-k short-list within it (default k=3). The candidate range follows paper App. D.4 — "the union of layers where CohH, CohV, and CohD are near peak" — after dropping the last --recommend-frac of layers (default 0.07; very deep or MoE models may need a smaller value, very shallow models tolerate larger). The paper notes that cross-model CohD rankings agree within range at Spearman ρ = 0.928, so the load-bearing output is the range itself — recommended and top_k are kept for back-compat and as representative picks; layers inside the range are reasonable starting candidates.

---

Installation

The framework needs only the standard scientific Python stack plus 🤗 transformers. Per-model dependencies (e.g., qwen-vl-utils) are imported lazily — you only need the ones for the models you actually probe.

Minimal install:

pip install torch transformers pillow numpy pandas tqdm \
            matplotlib seaborn scikit-learn

Model-specific extras:

pip install qwen-vl-utils                # qwen25, qwen3
# nvila: clone https://github.com/NVlabs/VILA and add to PYTHONPATH

A pinned environment we have tested with is provided in docs/requirements.lock.txt.

---

Data

The default benchmark is EmbSpatial-Bench (TSV format, base64-encoded images). The TSV is mirrored on the Hugging Face Hub at ch-min/EmbSpatial-Bench-tsv:

huggingface-cli download ch-min/EmbSpatial-Bench-tsv \
    EmbSpatial-Bench.tsv --repo-type dataset --local-dir ./data

Then point --data_path ./data/EmbSpatial-Bench.tsv (this is also the script's default). The loader normalises category aliases (under → below) and filters Unknown distance answers automatically.

You can use any other benchmark by producing a TSV with the same columns (index, question_id, category, question, image, answer, A, B, C, D).

SpatialTunnel benchmark

SpatialTunnel is the Blender-rendered diagnostic benchmark introduced in the paper to isolate spatial-shortcut biases by removing background and perspective confounds. The benchmark data is available on the Hugging Face Hub at cubec/spatialtunnel.

It is organized into three TSV configs:

All configs are tab-separated files with base64-encoded PNG images, spatial questions, and ground-truth answers. The generation code lives in a separate repository:

👉 cube-c/spatialtunnel-dataset-gen

---

Quick start

1. Run probing for one checkpoint

python probing.py \
    --model_type qwen25 \
    --scales vanilla \
    --data_path /path/to/EmbSpatial-Bench.tsv

This will:

1. Load the model on cuda.
2. Register a forward hook on every transformer layer.
3. Run 2 × N forwards (original + swapped) for up to 200 pairs per category.
4. Save raw vectors, axis coherence, delta heatmaps, and per-layer plots under results/saved_data/<model>_<scale>/.

Useful flags:

2. Cross-scale / cross-model comparison

After running each checkpoint separately, regenerate comparison plots from saved data — no model loads needed:

python probing.py --model_type qwen25 --merge

3. Multi-GPU shell wrappers

Helper scripts in scripts/ launch all scales of one family in parallel on assigned GPUs, e.g., scripts/run_qwen.sh.

---

Output layout

results/
  saved_data/
    qwen25_vanilla/
      npz/  vectors_vanilla.npz                 # raw embeddings + Δ vectors per layer
      csv/  delta_similarity_vanilla_L<k>.csv   # 6×6 matrices, one per layer
      json/ axis_coherence_vanilla.json         # mean / std / n per (group, layer)
            vd_ei_vanilla.json                  # VD-Entanglement Index per layer
      plots/
        metrics/axis_coherences.png             # axis coherence trajectory (all groups)
        metrics/vd-ei.png                       # VD-EI trajectory
        heatmap/...
        pca/...
        pca_3d/...
  compare/<group>/plots/metrics/axis_coherences.png
  compare/<group>/plots/metrics/vd-ei.png
  logs/<model>_<scale>.log

The .npz files are the canonical artefact — every downstream analysis (VD-EI, custom probes, alternative dimensionality reductions, etc.) can be built directly from them. If you upgrade the script after running probing, re-render the new plots/metrics/ files in place with:

python probing.py --model_type qwen25 --rebuild-metrics

---

Adding a new model

Two steps. Inherit from BaseHiddenStateExtractor and implement four small methods, then register the class.

@register_model(
    model_type   = "my_vlm",
    checkpoints  = {"base": "org/my-vlm-7b"},
    display_name = "MyVLM-7B",
    plot_color   = "#17becf",
)
class MyVLMExtractor(BaseHiddenStateExtractor):
    def _load_model(self):
        self.model     = AutoModelForXxx.from_pretrained(self.model_path, ...)
        self.processor = AutoProcessor.from_pretrained(self.model_path)

    def _get_num_layers(self) -> int:
        return len(self.model.language_model.model.layers)

    def _get_layer_module(self, layer_idx: int):
        return self.model.language_model.model.layers[layer_idx]

    def extract_and_predict(self, image, question):
        self.hidden_states = {}                       # reset per-call
        inputs = self.processor(images=image, text=question, return_tensors="pt")
        out    = self.model.generate(**inputs.to(self.device), max_new_tokens=20)
        answer = self.processor.tokenizer.decode(out[0], skip_special_tokens=True)
        return self.hidden_states.copy(), answer      # hooks fill this in

Hook registration, cleanup, and prefill-only filtering are handled by the base class. The shared helper _find_llm_layers(...) covers most "language backbone is buried under several attribute paths" cases automatically.

Then:

python probing.py --model_type my_vlm --scales base

---

Citation

If you use this code or methodology, please cite our paper.

@inproceedings{min2026whyfarlooksup,
  title     = {Why Far Looks Up: Probing Spatial Representation in Vision-Language Models},
  author    = {Min, Cheolhong and Jung, Jaeyun and Lee, Daeun and Jeon, Hyeonseong and
               Su, Yu and Tremblay, Jonathan and Song, Chan Hee and Park, Jaesik},
  booktitle = {Proceedings of the European Conference on Computer Vision (ECCV)},
  year      = {2026},
}

---

License

This project is released under the MIT License.